Cloud native data API (Application Program Interface) construction method based on single data flow

Abstract

The invention relates to the technical field of networks, in particular to a cloud native data AP I construction method based on a single data stream, which comprises the following steps of: 10, constructing a multi-source heterogeneous data exchange framework; 20, building a streaming and batch integrated data processing framework; 30, performing data storage by adopting an Apache Hud i data lake; 40, AP I construction based on the FaaS platform is carried out; according to the method, the Apache Hud i data lake storage and the OLAP query engine based on the MPP architecture are adopted, most of data query requests can be dealt with, compared with a Restfu interface which is mostly used at present, the data requirements of various users can be effectively met, the AP I is established based on the FaaS platform, and the data query efficiency is greatly improved. The problem that all data APIs cannot be used and fault isolation cannot be realized when the service crashes due to the fact that the APIs are abnormal can be effectively solved.

Description

technical field [0001] The invention relates to the field of network technology, in particular to a cloud native data API construction method based on a single data stream. Background technique [0002] In the process of data analysis and utilization, the data itself, the analysis model based on the data, and the data application based on the data itself and the analysis model have great value of open sharing. The traditional means of open data sharing is data export, such as exporting to a local disk, FTP server, distributed file system, etc. This method is suitable for temporary, large-scale data exchange scenarios, but the platform will lose any information collection capabilities for exporting data, and cannot collect and audit data users, nor can it export models and data applications. It greatly limits the functional boundaries of data services. Therefore, in the scenario of creating an application ecosystem, interfaces, especially Restful interfaces, have become a m...

AI Technical Summary

Problems solved by technology

However, most of these technologies currently use a set of services to operate all interfaces, and it is impossible to dynamically adjust resources for each interface in a fine-grained manner. Moreover, when some API exceptions cause service crashes, all data APIs cannot be used, and failures cannot be achieved. isolation

Images