A system for implementing manifest in distributed database based on cache optimization

Abstract

The present invention provides a system for implementing manifest in a distributed database based on cache optimization, including: a manifest file, wherein the format of the manifest file is append-only written in units of chunks, each chunk is a series of version change logs, and each log carries an HLC timestamp; a table range manager cache, wherein the hotspot table is cached in a memory by using an LRU cache to manage the manifest file of each table in a distributed database application; a manifest management component, wherein a plurality of manifest files and a current file constitute the manifest management component; a savepoint cache, wherein each time a new manifest file is generated, the latest version information up to now is completely saved in the header of the new manifest file; and a changelist cache, wherein the latest and most recently used data snapshots are cached in the memory through the LRU cache on the basis of an external memory format.

Description

Technical Field

[0001] The present invention relates to the field of distributed databases, and more specifically, to a system for implementing manifests in distributed databases based on cache optimization. Background Art

[0002] Manifest is a storage technology for metadata in the database. It can be used to help the database manage multiple versions, restore consistency, and optimize queries with conditional filtering. It can even be used to implement time travel. However, this is a single-machine transaction, and the information in the manifest is too simple to support only recovery, and cannot help optimize queries with filter conditions during runtime.

[0003] In the stand-alone database version (such as rocksdb), the manifest records the transaction log of all changes. When the DB is restored, the database can be easily restored to a consistent state. When the latest manifest is successfully written to the disk, all manifest files except the latest version will be deleted. Although detailed information can be described in the manifest, which helps query optimization, it is difficult for the data lake itself to have transaction functions. Even metadata is a stand-alone transaction model, so it cannot support these functions under the distributed transaction model.

[0004] In data lake applications (such as IceBerg), manifest is also used to record data change logs (but it does not strictly provide the ACID characteristics of the database. Metadata can implement ACID through external configuration, but user data itself does not have relational transaction characteristics). In addition to the traditional change log, column statistics and schema change information are also recorded. The manifest can be used to find the snapshotid of a certain moment in the past, thereby realizing time travel. The manifest design is too complicated, which will cause the number of manifest files to explode, and there is performance loss in accessing the manifest in external memory, which has a great impact on performance. Summary of the invention

[0005] The purpose of the present invention is to provide a system for implementing manifest in a distributed database based on cache optimization.

[0006] The present invention aims to solve the problems existing in the prior art.

[0007] Compared with the prior art, the technical solution of the present invention and its beneficial effects are as follows:

[0008] A system for implementing manifest in a distributed database based on cache optimization includes: a manifest file, wherein the format of the manifest file is append-only written in chunks, each chunk is a series of version change logs, and each log carries an HLC timestamp; a table range manager cache, wherein the hotspot table is cached in memory by using an lru cache to manage the manifest file of each table in the distributed database application; a manifest management component, wherein a plurality of manifest files and a current file constitute the manifest management component; a savepoint cache, wherein each time a new manifest file is generated, the latest version information up to now is completely saved in the header of the new manifest file; a changelist cache, wherein the latest and most recently used data snapshots are cached in memory through the lru cache based on the external memory format; wherein the table range manager cache manages different table ranges, including but not limited to adding a table range and loading a table range from a file system; each table range uniquely corresponds to a manifest management component, wherein the manifest management component consists of one or more manifest files, and each manifest file consists of a savepoint cache and a group of changelist caches.

[0009] As a further improvement, the manifest file is named in a format of seq.manifest, wherein seq is an increasing sequence number.

[0010] As a further improvement, version change is one of the following: file creation, appending a record, and schema change record.

[0011] As a further improvement, the new version change is recorded in the memory. When the number of changes reaches a certain number or exceeds a certain size or a checkpoint is encountered, the records in the memory are packaged and written into the manifest file in the form of chunks; if the manifest file reaches the specified size, the current file is updated to point to the latest manifest file.

[0012] As a further improvement, recovery will first parse the current file, find the latest manifest file, find the most recent base version, and then apply version changes in sequence to quickly restore to the most recent consistent state.

[0013] The beneficial effects of the present invention are:

[0014] The present invention not only enriches various types of information in the manifest (which is helpful for query performance optimization, etc.), but also designs a set of efficient LRU Cache mechanisms, and implements an efficient manifest technology in the distributed database storage engine of the distributed transaction model; based on cache optimization, the present invention highlights modular split cache optimization rather than an overall large cache, which not only improves performance but also avoids the occupation of a large amount of memory.

[0015] The manifest structure is designed with complete information description and expansion capabilities. The manifest not only reports some information in the memory through lrucache to ensure performance, but also effectively reduces the size of the manifest file by caching some information. The cache mechanism not only controls the memory, but also avoids the IO overhead of reading the manifest during normal reading and writing, thereby improving performance. The implementation logic is optimized in memory control, and a complete cache mechanism is designed in the distributed database scenario, which improves the performance of the manifest itself and the efficiency of memory management. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the manifest structure provided by an embodiment of the present invention.

[0017] Figure 2 It is a schematic diagram of performing multi-level query pruning using the manifest cache mechanism for queries provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0020] Reference Figure 1 and Figure 2 As shown, a system for implementing manifest in a distributed database based on cache optimization includes: a manifest file, the format of the manifest file is append only in chunks, each chunk is a series of version change logs, and each log carries an HLC timestamp; a table range manager cache, which manages the manifest file of each table in the distributed database application using an lru cache and caches the hotspot table in memory; a manifest management component, a plurality of manifest files and a current file constitute the manifest management component; a savepoint cache, each time a new manifest file is generated, the latest version information so far is completely saved in the header of the new manifest file; a changelist cache, based on the external memory format, caches the latest and most recently used data snapshots in memory through the lru cache; wherein the TableRange Manager cache manages different Table Ranges (shards), including but not limited to adding Table Ranges and loading Table Ranges from the file system; each Table Range uniquely corresponds to a manifest management component, which consists of one or more manifest files. Each manifest file consists of a savepoint cache and a set of changelist caches. Using LRU cache technology to manage thousands of Table Ranges (shards) not only improves performance, but also keeps memory well controlled. A series of cache technologies have been used to achieve a good integration of performance and memory management.

[0021] The manifest file is named in the format of seq.manifest, where seq is an increasing sequence number.

[0022] A version change is one of the following: file creation, an append record, a schema change record, or other information. It is flexible and scalable. This separates the logic of removing invalid data from deleting old data files. When a new version of data is generated, the old version of data is not deleted, thus providing the ability to query old data for time travel.

[0023] Record new version changes in memory. When the number of changes reaches a certain number or exceeds a certain size or a checkpoint is encountered, the records in memory are packaged and written to the manifest file in the form of chunks. If the manifest file reaches the specified size, the current file is updated to point to the latest manifest file.

[0024] Recovery will first parse the current file, find the latest manifest file, find the most recent base version, and then apply version changes in sequence to quickly restore to the most recent consistent state.

[0025] To alleviate memory issues, the changelist cache will periodically flush to external memory in the form of chunks, or clean up expired and unused snapshots, or it may swap the entire manifest to external memory.

[0026] Table range manager cache: Distributed database applications may usually have tens of thousands of tables, each with one or more ranges (shards). These tables cannot all be stored in memory. By using LRU cache to manage the manifest of each table, hot tables can be cached in memory as much as possible, improving performance without the problem of manifest memory being out of control.

[0027] Manifest stats data cache: Traditional manifest files are large, which increases IO costs.

[0028] This patent not only reduces the file size of the manifest itself by dynamically managing stats information of different levels of data files at runtime, but also provides efficient query tailoring support for hot query blocks or files.

[0029] When the query is executed, after finding the corresponding table version snapshot from the cache, the query first filters out the range that must have no data at the plan level according to the query conditions (for example, where col1>100and col1<1000), and then dispatches the trimmed plan to different Query Executors (QEs) for execution. Before the actual execution, QEs will perform file-level trimming again based on the stats data in the manifest cache to filter out unnecessary data files or data blocks, thereby reducing the amount of data that needs to be actually read and transmitted, and improving query performance.

[0030] The above embodiments are only used to explain the technical solutions of the present invention rather than to limit them. Those skilled in the art should understand that any modification and equivalent substitution that does not depart from the spirit and scope of the present invention should fall within the protection scope of the claims of the present invention.

Claims

1. A system for implementing manifest in a distributed database based on cache optimization, characterized in that: include: The manifest file is in the format of append only with chunks as the unit. Each chunk is a series of version change logs, and each log has an HLC timestamp. Table Range Manager cache, which manages the manifest file of each table in the distributed database application using LRU cache to cache hot tables in memory; Manifest management component, several manifest files and a current file constitute the manifest management component; Savepoint cache: every time a new manifest file is generated, the latest version of the information will be completely saved in the header of the new manifest file; Changelist cache, based on the external memory format, caches the latest and most recently used data snapshots in memory through lru cache; Among them, the Table Range Manager cache manages different Table Ranges, including but not limited to adding Table Ranges and loading Table Ranges from the file system; each Table Range uniquely corresponds to a manifest management component, the manifest management component consists of one or more manifest files, and each manifest file consists of a savepoint cache and a group of changelist caches.

2. A system for implementing manifest in a distributed database based on cache optimization according to claim 1, characterized in that: The manifest file is named in the format of seq.manifest, where seq is an increasing sequence number.

3. The system for implementing manifest in a distributed database based on cache optimization according to claim 1, characterized in that: A version change is one of the following: file creation, appending a record, or schema change record.

4. The system for implementing manifest in a distributed database based on cache optimization according to claim 1, characterized in that: Record new version changes in memory. When the number of changes reaches a certain number or exceeds a certain size or a checkpoint is encountered, the records in memory are packaged and written to the manifest file in the form of chunks. If the manifest file reaches the specified size, the current file is updated to point to the latest manifest file.

5. The system for implementing manifest in a distributed database based on cache optimization according to claim 1, characterized in that: Recovery will first parse the current file, find the latest manifest file, find the most recent base version, and then apply version changes in sequence to quickly restore to the most recent consistent state.

Images