Cross-regional task scheduling method and system based on big data

Abstract

The invention discloses a cross-region task scheduling method and system based on big data. The cross-region task scheduling method comprises steps of a user issuing a structured query language; analyzing the structured query language to generate a syntax tree; adopting a cross-domain scheduling engine to disassemble the syntax tree according to the metadata information; generating and distributing a plurality of logic execution plans executed on a total data platform or a sub-data platform, receiving a cluster of the logic execution plans, acquiring metadata stored in the cluster by a cross-domain scheduling engine, and acquiring position information of all data blocks from the metadata to generate a logic execution plan finally executed in the cluster; The cross-domain scheduling enginedistributes a logic execution plan to the data nodes, the data analysis engine on the data nodes receiving the logic execution plan performs data reading and calculation, and the cluster summarizationcalculation generates a preliminary summarization result and sends the preliminary summarization result according to the sending position information; And the cluster receiving the preliminary summarization result performs secondary summarization calculation on the data to generate a final query result and returns the final query result to the user. Mass data transmission is avoided, the bandwidth is saved, and the cost is reduced.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of data processing, and in particular to a large data-based cross-regional task scheduling method and system. Background technique [0002] At present, China Unicom and China Telecom will establish big data platforms in various provinces, and then upload the stored data files to the big data platform of the headquarters through the network every day, and then do data auditing and data analysis at the headquarters, usually using Hive ( A data warehouse tool based on Hadoop, which can map structured data files into a database table, and provide a simple sql query function, which can convert sql statements into MapReduce tasks for operation) or Spark (designed for large-scale data processing The fast and general computing engine designed can be used to complete various operations, including SQL query, text processing, machine learning, etc.) and other Mpp tools. Generally, it is neces...

AI Technical Summary

Problems solved by technology

[0011] First, a large amount of data is transmitted to the headquarters every day, occupying a large amount of bandwidth, and the cost of VPN is relatively high

[0012] Second, save multiple copies of the same data, wasting disk space

[0013] Third, regardless of the headquarters or the province, when doing data analysis, you can only use the computing resources of your own big data platform. Even if the resources are free, you cannot use each other's computing resources, resulting in waste of resources

[0014] Fourth, when personnel from various provinces do data analysis, they can only see the data of their own province, but not the data of the other party, and cannot do horizontal and cross-provincial comparative analysis of data

[0015] Fifth, a large number of personnel are required for support, and the labor cost is high

[0016] Sixth, the establishment of a super-large-scale hadoop cluster at the headquarters to carry the data of the whole country, the construction cost is very high

[0017] Seventh, the data delay of the headquarters accessing the provinces is very high. The headquarters can only access the data of the previous day in each province, and cannot access the data of the provinces in real time.

Images