Cache-based big data processing dimension table storage and calculation system and method thereof

Abstract

The invention discloses a big data processing dimension table storage and calculation system based on cache and a method thereof. The big data processing dimension table storage and calculation system comprises a data storage module, a dimension table configuration module, a Redis connection management module, a data type conversion module, a primary key processing module, a common field processing module, a data write-in module, a data association module and an Flink integration module. The data processing speed and throughput can be obviously improved through Redis, the problem of data processing delay is solved, and the problem of memory overflow caused by too large associated data can be eliminated through a Redis dimension table; and in addition, an Flink SQL mode can be provided to operate, read and write Redis data, Redis services of a single machine, a cluster, a sentry and an agent mode are packaged, a service interface and a big data development technology stack are unified, and the big data development convenience is improved.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a cache-based big data processing dimension table storage and calculation system and a method thereof. Background technique [0002] With the vigorous development of information technology, under the general trend of digital transformation in all walks of life, the data scale of enterprises is getting larger and larger, and traditional data processing technologies can no longer meet the demands of real-time processing of massive data. Thanks to hadoop, Spark With the continuous development and evolution of big data processing technologies such as , Flink, etc., data is processed with higher throughput and lower processing delay, so fast processing speed is the key to maximizing the value of extracted data. Technological iteration has also made data processing easier and more convenient. Big data development technology has developed from Java and Scala to Python and SQL. At pres...

AI Technical Summary

Problems solved by technology

Commonly used dimension table storage services in the industry include HBase, MySQL, Redis, etc. Among them, HBase and MySQL are officially supported by Flink, providing SQL-style data writing and association solutions. The advantage of MySQL as a dimension table lies in its complete data format and flexible association Semantic, but limited by the data read and write performance of MySQL, it only supports data association in the case of small data volumes, and cannot support large data volumes and high concurrent reading and writing; while HBase adopts the LSM data structure, and HDFS is used for the underlying storage. It has better performance for writing massive data, but the reading performance is not very high; Redis is a memory-based, distributed key-value pair storage database developed in C language, using memory storage to avoid low-speed disk operations, The data reading and writing of Redis is very fast, and the distributed feature allows Redis to support concurrent access to large amounts of data. However, since Redis is not officially supported by Flink, most of the industry uses Java code when using Redis as a dimension table, calling Redis API to operate data read and write, or write UDF, use Flink SQL call

Both of these solutions require Java coding and do not support code reuse, and the operation is cumbersome, which is not conducive to the unification of the technology stack and the maintenance of code and tasks

Images