Unlock instant, AI-driven research and patent intelligence for your innovation.

Big data multi-dimensional analysis and calculation efficiency improvement method and system

A multi-dimensional analysis and computational efficiency technology, applied in the field of big data analysis, can solve problems such as low query efficiency, and achieve the effect of improving operating efficiency, solving high latency, and improving service quality

Pending Publication Date: 2019-09-10
SHANGHAI MUNICIPAL ELECTRIC POWER CO +1
View PDF4 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art, provide a method for improving the computational efficiency of multi-dimensional analysis of big data, and solve the technical problem of low query efficiency in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data multi-dimensional analysis and calculation efficiency improvement method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment

[0047] The above method is now illustrated by an example as follows:

[0048] The implementation steps include: ①Start engine service; ②Business Cube configuration.

[0049] 1. Based on the big data platform environment, start this service. Configure environment variables such as JDK, configure environment variables such as Hive and HBASE in the big data environment, and then run the startup script to start the engine service.

[0050] 2. Business Cube configuration includes the following steps:

[0051] 2.1 Create a new project

[0052] First add a new multidimensional data analysis (OLAP) project, fill in the project description information and submit it.

[0053] 2.2 Synchronize Hive data tables

[0054] Load the metadata of the Hive table, select the table to be synchronized, perform synchronization, and add the source data to the engine management.

[0055] 2.3 Create a new Cube

[0056] Add Cube, and then perform Cube design. Cube design mainly includes the followi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a big data multi-dimensional analysis and calculation efficiency improvement method, which comprises the following steps of determining a fact table and a dimension table of acertain business scene and the statistical index types under different dimensions; associating the fact table with the dimension table to construct a data model; designing a data Cube according to thedata model; aiming at Cube, calculating to obtain the statistical indexes of different dimensions; storing the statistical index value in an HBase; and when the statistical indexes of the service scene under different dimensions are queried, directly querying the statistical index values stored in the HBase. According to the method and the system, the data Cube is quickly constructed on the basisof the computing capacity and the storage capacity of the big data platform, and the data cube is stored in the Key-value database HBase, so that the problem of high OLAP delay in big data is effectively solved, the operation efficiency is improved, and the service quality of the service application is improved.

Description

technical field [0001] The invention belongs to the technical field of big data analysis, and in particular relates to a method and system for improving computing efficiency of big data multi-dimensional analysis. Background technique [0002] With the advancement of power grid operation informatization and digitization, the amount of accumulated data continues to grow. There is an urgent need for multi-dimensional analysis of the accumulated massive data. However, the traditional OLAP (Online Analytical Processing) technology is difficult to meet the efficiency requirements, and it is necessary to realize OLAP analysis of billions of data based on the big data platform. [0003] However, it is difficult to guarantee computing efficiency by directly adopting big data Map / Reduce technology. For example, directly adopting Hive data warehouse has the disadvantages of high delay and high computing resource usage. Especially when generating monthly, quarterly, and annual report...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/21G06F16/2453G06F16/2458G06F16/28
CPCG06F16/212G06F16/2453G06F16/2462G06F16/283
Inventor 张琦孙立华刘士进孟庆强郑浩泉杨志刘铭钱亚康周洁
Owner SHANGHAI MUNICIPAL ELECTRIC POWER CO