Rapid calculation method adaptive to big data indexes

Abstract

The invention provides a method adaptive to fast calculation of big data indexes. The method comprises the following steps: synchronizing data needing to be treated into a hive cluster; converting the configuration of the page into an SQL statement through an SQL converter; the atomic indexes, the derived indexes, the composite indexes and the user-defined indexes are converted into SQL statements through an SQL converter; and calculating the index data by using a manner of integrating trino with hive. According to the scheme, the method capable of being adaptive to fast calculation of the big data indexes is provided, the problem that performance is insufficient when an existing index calculation method executes complex index calculation is solved, index calculation efficiency is improved, and the calculation efficiency is improved through step description in the embodiment. Rapid calculation can be achieved in index calculation, and friendly interaction experience is improved. The calculation method completely depends on memory operation, the performance is effectively improved, the problem of large disk I / O consumption caused by shuffle of a hive bottom layer is solved, and server resource consumption is reduced.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method for fast calculation of indicators adapted to big data. Background Art

[0002] With the development of society, SQL, as a universal language for data analysis, plays an especially important role in processing massive data sets. Traditional RDBMS often cannot cope with TB or even PB-level data due to performance bottlenecks and scalability limitations. Therefore, massive data is often synchronized to hive for index calculations, and the horizontal expansion of the cluster is used to share the query pressure and achieve high-performance query, which to a certain extent solves the problem of fast calculation of massive data.

[0003] However, in the existing technology, Hive is more suitable for ETL processing. When it is applied to the field of indicator calculation, the following problems will occur: 1. Due to the underlying implementation logic of Hive, each query statement is parsed into a map reduce task, which results in each indicator calculation having to wait for the task to end, resulting in a long response time and poor interactive experience. 2. High I / O overhead. Since the underlying layer of Hive is a Hadoop cluster, it requires high disk I / O operations and high resource overhead. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method for fast calculation of big data indicators to solve the problems raised in the above-mentioned background technology. The present invention makes up for the problem of insufficient performance of the existing indicator calculation method when executing complex indicator operations, improves the indicator calculation efficiency, and saves server resource consumption.

[0005] In order to achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: 1.

[0006] Further, the beneficial effects of the present invention are:

[0007] 1. The present invention provides a method that can be adapted to the rapid calculation of big data indicators. This solution makes up for the problem of insufficient performance of existing indicator calculation methods when executing complex indicator operations, and improves the efficiency of indicator calculation.

[0008] 2. This method for fast calculation of big data indicators completely relies on memory operations, which effectively improves performance and solves the problem of high disk I / O consumption caused by the shuffle at the bottom of Hive, saving server resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 The present invention is a flowchart of a method for fast calculation of indicators adapted to big data. DETAILED DESCRIPTION

[0010] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0011] See also Figure 1 The present invention provides the following technical solution: a method for fast calculation of adaptive big data indicators, the solution comprising:

[0012] 1. Synchronize the data that needs to be managed to the hive cluster

[0013] 2. Convert the page configuration into SQL statements through the SQL converter.

[0014] Atomic indicators, derived indicators, composite indicators, and custom indicators are converted into SQL statements through the SQL converter.

[0015] 3. Use trino to integrate hive to calculate indicator data.

[0016] The above solution provides a method that can adapt to the rapid calculation of big data indicators, which makes up for the problem of insufficient performance of existing indicator calculation methods when performing complex indicator operations and improves the efficiency of indicator calculation.

[0017] In this embodiment, the following calculation process is also provided for explaining the above scheme, and the specific steps are as follows:

[0018] 1. Synchronize data to the data warehouse. This is data preparation. In this embodiment, ETL tools are used to synchronize data, synchronizing external data of other data source types to the data warehouse, so as to facilitate subsequent indicator processing.

[0019] 2. Configure 4 types of indicators through the page, including atomic indicators, derived indicators, composite indicators, custom indicators, etc. After configuration, use the platform SQL converter to convert the indicators into executable SQL statements.

[0020] 3. Use trino to integrate hive, replace the query engine with trino, and then use trino's characteristics of being completely based on memory computing to perform efficient indicator calculations and return the calculation results to the page.

[0021] Through the steps described in the above embodiments, it is possible to achieve fast calculation of indicators and improve friendly interactive experience.

[0022] This calculation method completely relies on memory operations, which effectively improves performance and solves the problem of high disk I / O consumption caused by the underlying shuffle of Hive, saving server resource consumption.

[0023] The basic principles and main features of the present invention and the advantages of the present invention are shown and described above. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

[0024] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for fast calculation of big data indicators, characterized in that: The following steps are involved: Step 1: Synchronize the data that needs to be governed to the data warehouse; Step 2: Convert the page configuration into SQL statements; Step 3: Use trino to query and calculate indicators; Step 4: Return the result to the page.

2. The method for fast calculation of adaptive big data indicators according to claim 1, characterized in that: In the step 1, an ETL tool is used to synchronize data to synchronize external data of other data source types to the data warehouse.

3. The method for fast calculation of adaptive big data indicators according to claim 1, characterized in that: The page configuration includes four types of indicators: atomic indicators, derived indicators, composite indicators, and custom indicators.

4. The method for fast calculation of adaptive big data indicators according to claim 3 is characterized by: After configuration, use the platform SQL converter to convert indicators into executable SQL statements.

5. The method for fast calculation of adaptive big data indicators according to claim 1, characterized in that: In step 3, the query engine is replaced with trino by integrating hive with trino.

6. A method for fast calculation of adaptive big data indicators according to claim 5, characterized in that: Trino is fully based on memory computing to perform efficient indicator calculations.

7. The method for fast calculation of adaptive big data indicators according to claim 1, characterized in that: The conversion process in step 2 is completed through the SQL converter.

8. The method for fast calculation of adaptive big data indicators according to claim 1, characterized in that: The configuration of the page includes atomic indicators, derived indicators, composite indicators, and custom indicators.

9. The method for fast calculation of adaptive big data indicators according to claim 1, characterized in that: In step three, trino is used to integrate hive to calculate the indicator data.

Images