Method and device for mixed use of memory and temporary table space in presto computing nodes
A technology for computing nodes and temporary tables, applied in the computer field, can solve problems such as insufficient cluster memory, and achieve the effect of solving insufficient memory, improving query performance and concurrency
Active Publication Date: 2020-03-27
厦门见福连锁管理有限公司
View PDF2 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0011] The present invention adds the temporary table space function in the Presto application, and uses the method of mixed use of memory and temporary table space, which effectively solves the problem of insufficient cluster memory, and is simple and easy to use, which can greatly improve the cluster query performance and concurrency
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0016] Exemplary embodiments of the present invention are described below, and various details of the embodiments of the present invention are included to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various modifications and changes of the embodiments described herein can be made without departing from the scope and spirit of the invention.
[0017] Generally speaking, Presto data processing requires that the memory size can accommodate all the data involved in the calculation. Otherwise, the calculation will be very slow or a memory error will occur, resulting in the failure of the calculation task. At the same time, it is too dependent on the memory size, which also leads to the concurrency of Presto. Sex is not high.
[0018] Based on the shortcomings of existing technical solutions, we propose a method of adding temporary table space and memory to each computing node of Presto. The...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a method for the mixed use of a memory and a temporary table space at a Presto computational node. The method comprises the following steps: transmitting computational data to the Presto computational node; and if the required memory exceeds the free memory of the Presto computational node or exceeds the maximum memory allowed to be used by a single computational task on the Presto computational node, using the temporary table space. According to the method, a temporary table space function is added in a Presto application; and by using the method for the mixed use of the memory and the temporary table space, the problem of insufficient cluster memory is effectively solved; and the method is simple and easy to use, so that the cluster query performance and concurrency can be greatly enhanced. The invention furthermore provides a corresponding device.
Description
technical field [0001] The invention relates to computer technology, in particular to a method and a device for mixed use of memory and temporary table space in Presto computing nodes in big data applications. Background technique [0002] Presto is a distributed SQL query engine applied to big data. All data processing and transmission are based on memory and network. The calculation process is done in one go, no stages, no intermediate temp stage, avoiding unnecessary I / O and Delay overhead, so the overall query efficiency is nearly 10 times higher than Hive. [0003] During the calculation process, Presto needs to split and load all the metadata involved in the calculation into the memory of each computing node to complete the calculation, such as querying, sorting, and storing intermediate result sets. Presto supports the parallel execution of multiple jobs, so it is necessary to set the maximum memory that a single computing task can use on each computing node server. ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/22
CPCG06F16/1847G06F16/2282G06F16/27
Inventor 戴东东吕信郭李明袁安峰
Owner 厦门见福连锁管理有限公司