A data storage and processing system and method for a distributed computing cluster

A distributed computing and data storage technology, applied in the direction of electrical digital data processing, special data processing applications, computing, etc., can solve the problems of easy cluster data channels, occupying cluster data processing capacity, low data processing efficiency, etc., to avoid clustering The effect of the data channel

Active Publication Date: 2020-04-03
ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] In a computer cluster, if the data files to be processed are large, it will occupy the data processing capacity in the cluster and consume the resources of the cluster, resulting in low data processing efficiency and easy cluster data channels

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data storage and processing system and method for a distributed computing cluster
  • A data storage and processing system and method for a distributed computing cluster
  • A data storage and processing system and method for a distributed computing cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] In order to make the purpose, features and advantages of the present invention more obvious and understandable, the following will use specific embodiments and accompanying drawings to clearly and completely describe the technical solutions protected by the present invention. Obviously, the implementation described below Examples are only some embodiments of the present invention, but not all embodiments. Based on the embodiments in this patent, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this patent.

[0040] This embodiment provides a data storage and processing system of a distributed computing cluster, such as figure 1 As shown, it includes: a storage unit 2, a control unit 5 and at least one calculation unit;

[0041] The control unit 5 is used to obtain the data file whose carrying capacity exceeds the threshold value from the storage unit 2, and the file is divided i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a distributed calculation cluster data storage processing system and method. The method comprises the steps that a data file with the bearing capacity larger than a threshold value is obtained from a storage unit, the file is split into at least one block file, and the number of block files corresponds to the number of computation units; an address of each block file is distributed to the corresponding computation unit; each computation unit reads block file data information according to an address of the received block file, the block files are computed, a computation result is written into a computation result file and sent to a control unit; the computation result file sent by the computation units is received, the computation result file is read, and the computation result file is stored into a storage unit. In a computer cluster, the control unit obtains the data file with the bearing capacity larger than the threshold value, the data file is split, and processing is performed through the computation units. The large file is split, blocking processing is performed, the processing efficiency is improved, and processing resources in the cluster are sufficiently utilized.

Description

technical field [0001] The invention relates to the field of computer file processing, in particular to a data storage and processing system and method of a distributed computing cluster. Background technique [0002] At present, computer cluster technology has been widely used in many IT fields such as big data, cloud computing, and high-performance computing. Distributed file system (distributed file system) is a common component of computer clusters, especially high-performance computing cluster systems. Through distributed file systems, computing units in the cluster can share data located on other units. [0003] In a computer cluster, if the data files to be processed are large, it will occupy the data processing volume in the cluster and consume the resources of the cluster, resulting in low data processing efficiency and easy clustering of data channels. SUMMARY OF THE INVENTION [0004] In order to overcome the above deficiencies in the prior art, the present inv...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08G06F16/182
CPCG06F16/182H04L67/1097
Inventor 王志华
Owner ZHENGZHOU YUNHAI INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap