High-efficiency processing method and system for big data

Abstract

The invention relates to a high-efficiency processing method for big data, which comprises the following steps that a data node receives data to be stored; the data node stores the data, an index is simultaneously created according to a business scenario and is stored in a memory, and the data is gradually stored in a disk by index curing; a user inputs a task request, and an SQL (Structured Query Language) engine implements rapid retrieval of the data according to the created index and outputs the data to a computational node; a task processing module of a management node executes task scheduling, applies for resources to a resource management module and determines a spare computational node, and the spare computational node processes the data; the finally processed data is shown for the user. The invention also discloses the high-efficiency processing system for the big data. According to the invention, all processing is executed concurrently; hardware equipment of a computer is utilized to the greatest extent; processing efficiency is greatly improved; the user can more rapidly obtain a processing result when a task is executed.

Description

technical field [0001] The invention relates to the technical field of computer big data application processing, in particular to an efficient processing method and system for big data. Background technique [0002] With the extensive development of large-scale projects such as safe cities and smart cities, data aggregation and data fusion have further developed, and the amount of data to be processed has reached TB and PB levels. The processing of large amounts of data has produced a series of practical problems. When relational databases face such a large amount of data, their technical architecture, processing capabilities, and processing methods are increasingly unable to meet user needs. [0003] The development of cloud computing and big data technology provides a good solution to the processing of massive data. The Hadoop framework system uses parallel computing (MapReduce) and distributed storage (HDFS) to realize the storage and calculation of large amounts of data....

AI Technical Summary

Problems solved by technology

However, since the distributed storage (HDFS) does not support the direct processing of structured query statements (SQL), the data in the distributed storage (HDFS) is difficult to be processed directly, and the computing tasks need to be transformed into the parallel computing MapReduce framework in the end. Execution, its management node (Jobtracker) has heavy tasks, low efficiency, and easily leads to single point of failure

Images