Data grouping method and device based on multiple engines, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the technical field of big data, and relates to a data grouping method based on multiple engines, which comprises the following steps: initiating data grouping through Hadoop according to a request of a user side; receiving a data packet calculation instruction sent by the user side according to the data groups; judging whether the data packet calculationinstruction is of a high-computation task type or not through ElasticSearch (ES); if the data packet calculation instruction is a high-operand task type, judging whether the operand of the data packet exceeds an operand threshold value or not through ES according to the data packet calculation instruction; if the operand does not exceed the operand threshold value, achieving the operation throughES, and outputting an operation result; and if the operand exceeds the operand threshold, realizing operation through spark and outputting an operation result. The invention also relates to a blockchain technology. The data packet is stored in the blockchain. According to the method, the tasks with the small task amount are processed through the ES engine, the tasks with the high calculation amount are processed through the spark engine, and therefore the clustering efficiency is improved.

Description

technical field [0001] This application relates to the field of big data technology, and in particular to methods, devices, computer equipment and storage media based on multi-engine data grouping Background technique [0002] A query engine is a product based on data sources including SQL databases or SQL-like databases for data processing, data extraction, and query work. It usually needs to deal with large-scale and high-concurrency data processing tasks. Existing query engines use multi-dimensional Retrieval: Divide a query into multiple subtasks, and process these subtasks concurrently. During the processing, a certain degree of pre-storage is required, and then the results are further extracted or combined to output query or read results. When the query dimension of the class query engine increases, it will occupy a large amount of computing resources, and the occupancy of storage resources will also increase geometrically. If the amount of processed data increases, th...

AI Technical Summary

Problems solved by technology

However, when the amount of calculation increases, due to the large amount of data, the response to a large number of deep page-turning deep queries and aggregation queries is slow; the work performance is unstable when the data is updated frequently; the cost of data writing is high, so when the amount of data is large, it will Affect the stability, thereby affecting the query efficiency of data grouping

Images