Heterogeneous cluster data processing method and system based on multiple data centers and electronic equipment
A data processing system and data center technology, applied in electrical digital data processing, special data processing applications, file systems, etc., can solve problems such as affecting computing time, large IO consumption, increasing development difficulty and development cycle, etc. Effects of efficiency, faster processing, efficient resource fusion and task allocation
Pending Publication Date: 2020-05-12
SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
View PDF3 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
The disadvantage of this solution is that since the data is stored in different clusters, inter-cluster communication and transmission will occur during the calculation scheduling process. Therefore, a large amount of IO consumption will be generated during the calculation process and affect the overall calculation time
In addition, this solution is to modify the internal code of all clusters or add functions to improve performance and run on multiple machines, which will increase the development difficulty and development cycle, and is not applicable to all versions
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0037] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.
[0038] see figure 1 and figure 2 , figure 1 is a flowchart of the multi-data center-based heterogeneous cluster data processing method according to the embodiment of the present application, figure 2 It is the overall architecture diagram of the data processing system. The multi-data center-based heterogeneous cluster data processing method according to the embodiment of the present application includes the following steps:
[0039] Step 100: Select a computing area to obtain data of the data center in the computing area;
[0040] In step 100, the calculation ar...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to a heterogeneous cluster data processing method and system based on multiple data centers and electronic equipment. The method comprises the following steps: constructing a manager, and performing task decomposition on data of a plurality of data centers in a computing area through the manager; receiving a task submitted by a user through a scheduler, performing segmentation operation on the task, and allocating the segmented small tasks to each corresponding data center; receiving a task through the data center and performing task calculation; and collecting a calculation result generated by each data center through a collector. According to the invention, a manager comprising a scheduler and a collector is constructed; tasks are received and scheduled through thescheduler, and calculation results generated by each node are collected through the collector, so that novel and efficient heterogeneous cluster resource fusion and task allocation based on multiple data centers are realized, the calculation efficiency is improved under the condition of eliminating inter-cluster communication, and the data processing speed is increased.
Description
technical field [0001] The present application belongs to the technical field of heterogeneous clusters, and in particular relates to a method, system and electronic device for data processing of heterogeneous clusters based on multiple data centers. Background technique [0002] In today's big data scenarios, due to geographic or other reasons, there will be data isolation and data storage. Therefore, a large amount of data transmission work is generated during the calculation process, resulting in an increase in processing time. However, in the current system pursuing high QOS (Quality of Service, quality of service), such a situation should be avoided as much as possible. [0003] In order to solve the above problems, Wang proposed in the paper a solution to build a Hadoop cluster with multiple computing nodes through the G-Hadoop framework structure. The solution is as follows: the entire hadoop cluster is arranged into a large cluster, and the slave node is controlled...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F9/50G06F16/13G06F16/182
CPCG06F9/5027G06F16/13G06F16/182Y02D10/00
Inventor 熊景盼王洋须成忠张锦霞
Owner SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI