Method and device for executing distributed computation tasks

Abstract

The invention discloses a method and device for executing distributed computation tasks. A specific implementation manner of the method comprises the following steps of: analyzing an expression of a distributed computation task so as to obtain an operator keyword; and executing the distributed computation task by utilizing an operator corresponding to the operator keyword on the basis of input parameters of the operator, wherein the input parameters comprise at least of the following items: a distributed data set which is stored in a distributed manner and consists of at least one data element and a distributed key value pair set which is stored in a distributed manner and consists at least one key value pair, and values of key value pairs in the distributed key value pair set are distributed data sets or distributed key value pair sets. Through the implementation manner, the operation carried out on nested data sets is realized, so that the distributed computation tasks of operating the distributed data sets and operating the grouped data of the distributed data sets after the grouping are supported, and the code reusability is realized.

Description

technical field [0001] The present application relates to the field of computers, in particular to the field of distributed computing, and in particular to methods and devices for executing distributed computing tasks. Background technique [0002] In the development of distributed computing programs, users (such as engineers writing distributed computing programs) can use the interfaces provided by the distributed computing framework to write distributed computing programs. [0003] The interfaces provided by existing distributed computing frameworks mainly include native class interfaces (for example, MapReduce, a programming model for parallel computing of large-scale data sets), similar structured query language (SQL-Like, Structured QueryLanguage Like) class interfaces ( For example, the data warehouse infrastructure Hive built on the distributed system infrastructure Hadoop developed by the Apache Foundation) and the Resilient Distributed Dataset (RDD, Resilient Distri...

AI Technical Summary

Problems solved by technology

[0004] However, the existing distributed computing frameworks are generally inconvenient to use interfaces, especially the lack of the concept of nested data sets, resulting in complex logic expressions (especially the expression of grouping logic) and difficult code reuse.

Taking the SQL-Like class interface as an example, due to the lack of the concept of nested data sets in the SQL-Like class interface, it is impossible to describe some computing requirements

Take a common calculation requirement as an example: group data by a certain field, and take the top 10 records with the largest value of the group field in each group as an example, when the expression is "select field_1, field_2 from table_1group by field1order by field_2desc limit 10", actually the first 10 records are taken out globally, not the first 10 records for each group; when the expression is "select field_1, field_2from table_1group by field1limit 10order by field_2desc", the expression Does not conform to the SQL syntax, resulting in the inability of the SQL-Like class interface to describe some common computing requirements

Images