A double-layer Spark energy-saving scheduling method based on dynamic voltage frequency adjustment
A frequency adjustment and dynamic voltage technology, which is applied in the field of big data processing and energy efficiency, can solve problems such as resource consumption, resource consumption, and uneven distribution of running time, and achieve the effects of reducing energy consumption, time balance, and reducing idle running time
Active Publication Date: 2019-04-05
广州大鱼创福科技有限公司
View PDF7 Cites 4 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
And when the CPU is idle, there will also be resource consumption problems. Because of the uneven distribution of running time, the idle running CPU consumes a lot of resources.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
[0050] In order to reduce the idle running time of the node CPU and reduce the energy consumption of the application program during the Spark running process, the present invention proposes a two-layer Spark energy-saving scheduling method based on dynamic voltage frequency adjustment, such as figure 1 , including: the first layer, according to the different CPU and I / O requirements of different applications, by observing the resource usage characteristics of ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to the fields of big data processing and energy efficiency. The invention particularly relates to a double-layer Spark energy-saving scheduling method based on dynamic voltage frequency adjustment. The method comprises the steps that for different CPU and I / O requirements of different application programs, the number of application categories can be roughly judged in advanceby observing the resource use characteristics of different applications, and overall frequency reduction processing is conducted on the application programs; By considering the inherent characteristics of the Spark, the completion time of different tasks in the same stage in the application program running process is different, frequency reduction processing is carried out on specific nodes, the node idle running time is shortened, and the completion time balance of each Task is ensured; The task completion time balance in the Spark operation process is guaranteed, the node CPU idle operationtime is shortened, and the energy consumption generated by an application program in the Spark operation process is effectively reduced.
Description
technical field [0001] The present invention relates to the fields of big data processing and energy efficiency, in particular to a two-layer Spark energy-saving scheduling method based on Dynamic voltage and frequency scaling (DVFS). Background technique [0002] The increasing and large-scale deployment of high-performance computing clusters has brought about huge energy consumption. The statistical results of the literature show that the power consumption of global Internet data centers may reach 30 billion watts, which is equivalent to the power supply of 30 nuclear power plants. However, only 6% to 12% of the power consumption of the data center is used for website computing, and the rest is unnecessary consumption when maintaining the working state of the server. In addition, Greenpeace predicts that by 2020, the world's major IT operators will China's energy consumption will reach 2 trillion kWh, exceeding the combined energy consumption of Germany, France, Canada an...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F1/324
CPCG06F1/324Y04S10/50
Inventor 李鸿健马恩杰熊安萍蒋溢
Owner 广州大鱼创福科技有限公司