Method and device for predicting platform operation execution time and electronic equipment
A technology of execution time and operation, applied in the computer field, can solve the problems of low accuracy of complex operators, poor applicability, low prediction accuracy of interface-intensive operations, etc., to achieve accurate prediction, improve overall performance, and reduce complexity.
Pending Publication Date: 2021-01-29
SHANGHAI QIYUE INFORMATION TECH CO LTD
View PDF3 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0004] The execution time of complex operators in the distributed computing platform cannot be accurately predicted. In the prior art, a method for predicting the query execution time in the Hadoop distributed storage system has been studied. The nuclear canonical correlation analysis statistical model is used to correlate the query output size with the execution time. The performance of recent similar queries is used to predict job execution time. Experiments show that simple operators in the calculation process have certain reference value, and complex operators have low accuracy. At the same time, a hybrid model is proposed in the prior art to estimate different types of operators. It has good performance in relational databases, but its applicability in large-scale distributed computing platforms is poor, and it uses sample data to simulate the execution performance of different nodes in the cluster, and interface-intensive jobs with Shuffle overhead low prediction accuracy
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0048] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. However, example embodiments may be embodied in many forms, and this invention should not be construed as limited to the embodiments set forth herein. On the contrary, providing these exemplary embodiments can make the present invention more comprehensive and complete, and facilitate the full transfer of the inventive concept to those skilled in the art. The same reference numerals denote the same or similar elements, components or parts in the drawings, and thus their repeated descriptions will be omitted.
[0049] On the premise of conforming to the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.
[0050] In the description of the specific embodiments, the featu...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to the technical field of computers, in particular to a method and device for predicting platform operation execution time and electronic equipment, and the method comprises thesteps: extracting an operation data set from a target operation, and the operation data set comprises feature vector data and measurement index data; obtaining execution time of each stage of the target operation through a time prediction model; calculating the execution time of the target operation according to the execution time of each stage of the target operation; and automatically and dynamically adjusting and executing the execution plan of the target operation according to the predicted execution time of the target operation, and obtaining the adjusted execution plan of the target operation. According to the method, the execution time of the complex operator in the distributed computing platform can be accurately predicted, the job execution plan can be dynamically adjusted, the complexity of the computing process is effectively reduced, and the overall performance of the computing cluster is improved.
Description
technical field [0001] The invention relates to the field of computer technology, in particular to a method, device and electronic equipment for predicting the execution time of platform operations. Background technique [0002] The distributed computing platform can expand the cluster scale to thousands of nodes with the help of the core engine. CatalystOptimizer provides a rule-based and cost-based optimizer to push the computing power of the data warehouse to a new level. However, there are problems of ease of use and scalability on very large-scale datasets. The structured product information language or Dataset program is parsed into a logical plan before execution, and then an executable physical plan is generated. Different execution plans have a significant impact on performance. Have a great impact. [0003] Theoretical research on job execution time prediction of distributed computing platforms and technical practice have the following technical problems: [0004...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F16/2458G06F16/27G06F16/901G06F30/27
CPCG06F16/2465G06F16/2477G06F16/27G06F16/9024G06F30/27
Inventor 吴恩慈
Owner SHANGHAI QIYUE INFORMATION TECH CO LTD