Service platform integration method based on big data

Abstract

The invention discloses a service platform integration method based on big data. The method comprises the following steps: (1) acquiring multi-source heterogeneous data; (2) integrating the acquired multi-source heterogeneous data, and storing the integrated data in an Hbase database; (3) using Hive to perform ETL treatment on the integrated data which is stored in the Hbase database, and storing in the Hbase database, cleaning the data which is stored in the Hbase database, to obtain clean data, and storing the clean data in the Hbase database; (4) based on an Hadoop technology, performing modeling analysis on the obtained clean data, and storing an analysis result in an Hive database; (5) using a service-oriented architecture based on SOA to establish a data exchange and shared service bus, and then based on the service bus, establishing a data exchange architecture, and through the data exchange architecture, pushing the analysis result which is stored in the Hive database to a business application system database. The method effectively reduces communication cost and time cost, and improves data effective utilization rate.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a service platform integration method based on big data. Background technique [0002] Big data is the commanding height of the next generation of information technology in the information industry. The construction of smart cities has been put on the agenda in China. Smart cities contain a large amount of data and are a new generation of application technologies serving government affairs, enterprises and citizens. However, the existing smart cities Big data integration technology is still unable to serve government affairs and the general public in all aspects of the city, which is mainly due to the limitations of the following aspects: [0003] (1) The computational model of the complexity of big data is currently only analyzing the nature and exploring the laws of multi-source data, and there is no complete application method system; [0004] (2) There is less structured da...

AI Technical Summary

Problems solved by technology

[0002] Big data is the commanding height of the next generation of information technology in the information industry. The construction of smart cities has been put on the agenda in China. Smart cities contain a large amount of data and are a new generation of application technologies serving government affairs, enterprises and citizens. However, the existing smart cities Big data integration technology is still unable to serve government affairs and the general public in all aspects of the city, which is mainly due to the limitations of the following aspects:

[0003] (1) The computational model of the complexity of big data is currently only analyzing the nature and exploring the laws of multi-source data, and there is no complete application method system;

[0004] (2) There is less structured data and more unstructured data, and there is no advanced technology or means to deal with unstructured and semi-structured data;

[0005] (3) Exploring the description methods of big data complexity and uncertainty characteristics and the system modeling of big data are not yet perfect;

[0006] (4) The current big data mining is basically in the state of mining rough knowledge at the first time, and no more perfect secondary mining method has been found to provide decision-makers with intelligent knowledge to guide decision-making

Images