A stream data processing method based on sparksql and restapi

Abstract

The invention discloses a streaming data processing method based on a SparkSQL (structured query language) and a RestAPI (Rest Application Programming Interface). The method comprises the following steps: S1, initializing an algorithm in a to-be-packaged RestAPI algorithm library; S2, packaging the initialized algorithm into a UDF format; S3, registering the packaged UDF; S4, assembling a stream data calculation program; and S5, obtaining an SQL statement input by the user and executing the SQL statement. According to the method, on the basis of traditional SQL mass streaming data calculation,convenient complex algorithm calculation support is provided, and SQL can be conveniently used for achieving the complex algorithm calculation capacity of the mass streaming data. According to the invention, the real-time computing capability of SQL streaming data can be realized.

Description

technical field [0001] The invention belongs to the technical field of data service governance, and in particular relates to a service provision method based on Internet big data. Background technique [0002] In recent years, with the advent of the era of big data, data has shown explosive growth, and the scale of information has become increasingly large. Information data provides a solid information foundation for corporate decision-making. The impact and change brought about by the generation of massive data on society is unprecedented. , for enterprises, how to quickly and efficiently extract useful value from data has become a new challenge. Traditional data is dominated by structured data, but now data from social networking sites, e-commerce, and the Internet of Things are basically unstructured and semi-structured data. Traditional data can be effectively managed and managed with a relational database management system. use, and now the data is large, fast, and com...

AI Technical Summary

Problems solved by technology

Traditional data is dominated by structured data, but now data from social networking sites, e-commerce, and the Internet of Things are basically unstructured and semi-structured data. Traditional data can be effectively managed and managed with a relational database management system. use, and now the data is large, fast, and complex, which greatly exceeds the capabilities of traditional database software tools. According to the IDC report, it is estimated that by 2020, the global data usage will reach 35.2ZB. In the face of such a massive amount of data, processing The efficiency of data means whether the enterprise has the corresponding capabilities and competitiveness, so it is necessary to have an easy-to-use, highly reliable, and high-performance data analysis system

[0003] Publication No. CN108268639A "An Index Calculation Method in a Big Data Environment" provides an index calculation method in a big data environment, which separates the index calculation scheduling code and business calculation SQL, and combines data reading, preprocessing, The SQL for calculation and result storage is all put into the database, and a new index is added. Only a few SQL lines are needed to achieve the goal, basically realizing the need for zero code increase, but this solution can only use the existing SQL capabilities, and cannot expand the external The algorithm capabilities provided by RestAPI, that is, complex SQL algorithm calculations cannot be realized

Images