Distributed scheduling method of multi-source data analysis engine, computing node and distributed scheduling system of the multi-source data analysis engine

Abstract

The embodiment of the invention provides a distributed scheduling method of a multi-source data analysis engine, a computing node device, a readable storage medium, computing equipment and a distributed scheduling system of the multi-source data analysis engine. A large amount of communication cost is saved, the distributed processing efficiency of the multi-source data analysis engine is improved, and the method comprises the steps that a computing node with the highest scheduling index of the multi-source data analysis engine receives a query task; the computing node determines a sub-query task including an intermediate result set of the query task, and determines a storage node of the intermediate result set; the computing node computes a first time overhead of migrating the intermediate result set to the local, and computes a second time overhead of executing the sub-query task by the storage node; and the computing node selects whether to migrate the intermediate result set to thelocal and execute the sub-query task by the computing node or execute the sub-query task by the storage node according to a comparison result of the first time overhead and the second time overhead.

Description

technical field [0001] The present invention relates to the technical field of databases, in particular to a distributed scheduling method of a multi-source data analysis engine, a computing node device, a readable storage medium, a computing device, and a distributed scheduling system of a multi-source data analysis engine. Background technique [0002] The multi-source data analysis engine is a query language for docking with various data sources such as ElasticSearch, Mongo, and Mysql. Taking the Qingteng Structured Query Language (QSL) engine as an example, it can also provide pipeline syntax to filter query results, realizing continuous data analysis capabilities. The engine node saves the intermediate result set in the calculation process in the local Sqlite3 database. The engine itself does not have distributed capabilities, and can only call computing resources of a single node, which greatly limits the improvement of the computing power assembly and the concurrent ...

AI Technical Summary

Problems solved by technology

[0006] Yarn solves the problems of poor scalability, single point of failure, and limited to MR (MapReduce) computing framework of Hadoop's first-generation job scheduling system.

However, in terms of data management, the same HDFS as the first generation is still used. In the scenario of massive data above the PB (Petabyte, petabyte) level, this system plays an irreplaceable role as the only solution at present, but in There is still room for optimization in TB (Terabyte, terabyte) level data scenarios

Images