NiFi-based two-level cache data processing method and system

Abstract

The invention belongs to the technical field of data management, and discloses a NiFi-based two-level cache data processing method and system. The method comprises the steps of starting two-level cache server to monitor data signals; receiving a user operation request and converting it into a signal for sending to a server; sending the request buffer data to the secondary buffer; sending a SecondaryCache signal to a secondary buffer server; storing the cached data in SecondaryCache. The secondary cache client service sends the applySecondaryCache signal to the secondary cache server; The cached data of the SecondaryCache are copied into a new cache object and the new cache object pointer is pointed to the cache used for the query. The invention adds a two-level cache for temporarily storing cache data, and has atomicity to the operation of the original cache, so as to achieve the purpose of safely deleting the invalid cache.

Description

technical field [0001] The invention belongs to the technical field of data management, and in particular relates to a NiFi-based secondary cache data processing method and system. Background technique [0002] At present, the existing technologies commonly used in the industry are as follows: [0003] With the componentization of big data ETL, many ETL functions have been implemented to allow users to connect and configure the ETL process in the interface by dragging and dropping. NiFi is such a powerful ETL engine, and it has realized the requirements of many ETL components, which greatly reduces the threshold for users to perform ETL processing on data. As a caching component that is often used in the NiFi engine, it cannot meet most of the needs in the face of actual business scenarios. This is because when caching database query results, the deleted data in the database cannot be captured, resulting in discarded data. When performing cache query matching, outdated cac...

AI Technical Summary

Problems solved by technology

[0005] (1) When the existing technology caches the database query results, the deleted data in the database cannot be captured, resulting in the retention of the discarded data. When the cache query is matched, the outdated cache is matched, which affects the quality of business data

[0006] (2) When the existing technology uses the cache component, it cannot remove the invalid cache in time, which affects the use of other business components

[0008] The difficulty is: In actual projects, when querying the dictionary data of the database, the deleted data cannot be captured;

Images