A real-time metadata update method for spark-sql retrieval

Abstract

The invention relates to a metadata real-time update method for Spark-Sql retrieval. According to the method, when Spark Sql performs retrieval, used metadata is cached, and retrieved file metadata of data blocks is cached in a memory in a set form; when incremental information of the file metadata is obtained, whether retrieved metadata in a table which the incremental information belongs to exists in a cache is checked first; and if yes, incremental file metadata is added into the cached file metadata set in the table, and then incremental update of the retrieved metadata of Spark Sql is completed. The invention furthermore provides a processing method for addition of the retrieved metadata of Spark Sql. Through the processing method, the addition of the retrieved metadata can be processed in real time, and the real-time performance of a retrieval result is improved. Through the update method, frequent total update of the retrieved metadata can be avoided, therefore, metadata update time is shortened, and resource overhead during metadata update is reduced.

Description

technical field [0001] The invention relates to a method for updating metadata in real time for Spark Sql retrieval, which can improve the real-time performance and operation performance of Spark Sql when retrieving massive data, and belongs to the field of big data processing. Background technique [0002] With the continuous development of computer technology and the continuous improvement of informatization, the amount of data has grown rapidly, and the storage and application of massive data has also flourished. For example, in terms of network security, big data technology is used to analyze network attack behavior; in e-commerce, big data technology is used to analyze user shopping preferences or most popular products. Big data technology has played a positive role in building a conservation-oriented society and improving production efficiency. [0003] In massive data retrieval applications, Apache Foundation's distributed retrieval framework Spark Sql provides a Hiv...

AI Technical Summary

Problems solved by technology

When the metadata used for retrieval changes, the Spark Sql framework will discard all the original cache and reload the full amount of retrieval metadata, which will cause large time and resource overhead in the case of massive data

In business scenarios where metadata is frequently changed, frequent updates and retrieval of metadata cause performance bottlenecks in Spark Sql, which is a key issue in using Spark Sql for retrieval

With the continuous increase of data volume and the continuous development of big data technology, Spark Sql's native metadata update method has a performance bottleneck, and its efficiency directly affects business applications

Images