An archiving method for distributed databases

By combining differential metadata layers and asymmetric hierarchical indexes, the high cost and long latency issues caused by table structure changes in distributed databases are resolved, enabling efficient cross-version queries.

CN123115086APending Publication Date: 2026-11-17INNER MONGOLIA LINGHUI TECHNOLOGY DEVELOPMENT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611282310.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-24
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In distributed databases, when the business table structure evolves multiple times due to changes in the data definition language, existing technologies require a full rewrite of a large amount of archived cold data, resulting in high object storage call costs, high network bandwidth consumption, and difficulty in controlling query latency.

Method used

An incremental mapping of data definition language changes is recorded using a differential metadata layer. By generating virtual projection rows in memory and utilizing asymmetric hierarchical indexes, unified retrieval across multiple historical table structure versions is achieved, avoiding a full rewrite of cold data physical files.

Benefits of technology

It reduces object storage call costs and network bandwidth consumption, enables a unified query view across versions, avoids compatibility errors at the logical level, and reduces the number of network requests per query to a constant level independent of the number of candidate data blocks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115086A_ABST
    Figure CN123115086A_ABST
Patent Text Reader

Abstract

This invention discloses an archiving method for distributed databases, relating to the field of distributed database technology. Addressing the issues of discrepancies between archived cold data and the new structure after table structure changes, the high cost of existing full rewrite schemes, and the high number of retrieval accesses required by tree-index migration schemes, this method, upon detecting a table structure change, incrementally records the difference mapping and writes it to a metadata layer independent of the cold data physical file. During queries, the difference mapping is superimposed along the version chain to obtain a unified mapping function. Historical data is projected into the target structure in memory. Through the cooperation of the routing layer index in hot memory and the retrieval layer index in the same-source storage, only one read request is initiated to accurately locate the selected candidate data blocks. This achieves cross-version unified querying without rewriting the cold data physical file and reduces the number of retrieval network accesses to an approximately constant level.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of distributed database technology, specifically to an archiving method for distributed databases. Background Technology

[0002] To reduce total cost of ownership, distributed databases generally adopt a storage-compute separation and cold-hot data tiered archiving architecture: hot data is stored on local high-performance disks to support online transaction processing, historical cold data is converted into columnar immutable format and archived to object storage, and a secondary index structure is built to accelerate retrieval. Business systems usually undergo multiple data definition language changes during operation, and each change will generate a new table structure version, while the archived cold data remains fixed in the old structure before the change.

[0003] One existing approach is to read the affected historical archive files after detecting a structural change, re-encode them according to the new structure, and write them to object storage. At the same time, the old files are deleted and the metadata is updated. Due to the large volume of historical archive data, this approach requires re-encoding and physical file replacement line by line for a large amount of data, which significantly increases the cost of object storage calls and network bandwidth consumption. In addition, the rewriting process is time-consuming, and during this period, the archive data may be unqueryable or the query results may be inconsistent.

[0004] Another existing approach is to migrate the tree-like index structure originally used on the local disk to object storage. During retrieval, multiple nodes need to be accessed layer by layer from the root node to reach the leaf node. Since the latency of the first byte in object storage is significantly higher than that of the local disk, this access method causes the number of network requests for a single query to increase linearly with the height of the index tree, making query latency difficult to control. To address this, we propose an archive processing method for distributed databases. Summary of the Invention

[0005] The purpose of this invention is to provide an archiving method for distributed databases.

[0006] The technical problem this invention aims to solve is: in the case of a distributed database employing a storage-compute separation and cold / hot data hierarchical archiving architecture, when the business table structure evolves multiple times due to changes in the data definition language, how to achieve unified retrieval across multiple historical table structure versions without performing a full rewrite of the archived, immutable cold data physical files, and how to keep the number of network accesses generated during the cold data retrieval process at a low level, thereby eliminating the semantic differences between long-term archived data and dynamic table structure evolution.

[0007] To address the aforementioned technical problems, this invention provides an archiving processing method for a distributed database, comprising:

[0008] When a change in the data definition language of a business table in a distributed database is detected, a new table structure version number is assigned to this change, and the difference mapping set corresponding to this change is calculated. The difference mapping set is written incrementally to the difference metadata layer, which is independent of the cold data physical file. At the same time, the table structure version linked list is updated to record the sequential relationship between adjacent versions. The cold data physical file is an immutable columnar storage format file that has been archived to object storage. The process of writing to the difference metadata layer does not read or modify the cold data physical file.

[0009] Receive a query request for the cold data physical file, determine the historical table structure version corresponding to the historical data block involved in this query, and the target table structure version expected to be used in this query.

[0010] When the historical table structure version is inconsistent with the target table structure version, the table structure version is traced back from the historical table structure version to the target table structure version. The difference mapping sets corresponding to each version along the way are read in sequence and superimposed and merged according to the version order to obtain a unified mapping function.

[0011] Based on the unified mapping function, at least one of the following processes is performed on the physical column data in the historical data block in memory: offset relocation, type conversion, and default value filling, to generate a virtual projection row that conforms to the target table structure version, and the process of generating the virtual projection row does not modify the cold data physical file.

[0012] A routing layer index is maintained in the memory of the compute node. The routing layer index contains statistical information on the value range of each data block, member determination summary information, and table structure version timestamp. All data blocks are filtered based on the routing layer index to determine candidate data blocks. A read request is initiated for the determined candidate data blocks to obtain the retrieval layer index and data content corresponding to the data block. The retrieval layer index and the data content are organized in the same logical data block in the object storage and can be obtained as a whole in a single read request. There is no need to initiate separate network accesses for the index and data. The retrieval layer index and the cold data physical file are stored in the same object storage.

[0013] The system performs a retrieval of the data content locally based on the obtained retrieval layer index, locates the physical offset position of the target row, and performs table structure projection transformation by combining the located physical data with the generation method of the virtual projection row to obtain the query result.

[0014] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0015] 1. Because changes to the data definition language are recorded incrementally in the differential metadata layer, which is much smaller than the cold data, without touching the physical files of the cold data, this invention can eliminate the full rewrite of historical archived data that is forced to be triggered by changes in table structure. This reduces the call cost of object storage and network bandwidth consumption, and eliminates the problem that archived data is not searchable during the rewrite window.

[0016] 2. Because the present invention generates virtual projection rows in memory in real time during the query, projecting the historical heterogeneous table structure into the target table structure, it can achieve a unified query view across versions while keeping the underlying physical files immutable, thus avoiding compatibility errors at the logical level.

[0017] 3. By employing an asymmetric hierarchical design between the routing layer index in hot memory and the retrieval layer index in object storage, this invention reduces the number of network requests for a single query from the O(logn) level, which increases linearly with the height of the index tree, to an approximately constant level of O(1), which is independent of the number of candidate data blocks, compared to a tree-based index scheme that relies on multi-level node hierarchical network access. This quantitative comparison effect is a technical effect that is difficult for those skilled in the art to foresee based on existing tree-based index migration schemes. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the differential metadata generation process in an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the virtualization overwrite query processing flow in an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the asymmetric hierarchical index retrieval process in an embodiment of the present invention. Detailed Implementation

[0021] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0022] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] The technical solution of the present invention will be further described in detail below with reference to specific embodiments. The distributed database archiving processing method proposed in the present invention runs on a distributed database architecture with storage and computation separation. This architecture includes a computing node layer, a metadata management layer and an object storage archiving layer. The computing node layer is responsible for query parsing, query optimization and vectorized execution. The metadata management layer is responsible for maintaining the table structure version linked list, the difference metadata layer and the routing layer index. The object storage archiving layer is responsible for persistently saving immutable cold data physical files and retrieval layer indexes stored in the same source as them.

[0024] For ease of description, in this embodiment, the defined state of the database table structure at a certain moment is called a Schema version. Each time a data definition language change occurs (including adding columns, deleting columns, modifying field types, and modifying field lengths), a new Schema version is generated. The versions are ordered by version number and effective timestamp, and are linked together in the form of a table structure version linked list. The incremental metadata structure that is independent of the cold data physical file and is used to record the column-level differences between adjacent Schema versions is called a ghost column. It is carried in the ghost column SSTable layer set independently in the log structure merge tree and is isolated from the physical file carrying the cold data in terms of storage location. The processing mechanism that dynamically projects the cold data of the historical heterogeneous Schema into the target Schema structure during the memory scan phase of the query without modifying the underlying physical archive file is called virtualization overwrite.

[0025] I. Generation method of the difference metadata layer:

[0026] The metadata management layer continuously monitors data definition language requests initiated by business systems, captures their raw text and the identifiers of the target tables involved, performs semantic parsing on the captured statements, identifies the type of change, which includes at least one of adding a column, deleting a column, modifying a field type, and modifying a field length, and assigns a new schema version number to this change, denoted as . and set its effective timestamp. ,in Numerically greater than the currently effective historical version number. ,and Greater than Corresponding historical effective timestamp ;

[0027] Calculate the difference mapping set corresponding to this change based on the change type. The difference mapping set includes default values ​​for newly added columns, marker information for deleted columns, field type conversion rules, and the offset increment of each column relative to the old schema under the new schema. The offset mapping relationship between adjacent columns can be expressed as follows:

[0028] ;

[0029] In the above formula, Represents column New Schema Version Field offset below, Indicates the same column In historical schema versions Field offset below, Indicates the sequence number is The column offset increment introduced in this change will be a non-zero value if the column was added or its type length changed; otherwise, it will be zero. This represents the current column's sequence number within the table structure, with an upper limit for accumulation. This indicates that the offset increments of all columns preceding this column are accumulated;

[0030] For example, a certain business table in the schema version The following columns are included in order. (Integer, zero offset, four bytes in length) (String type, offset of four, length of twenty bytes) and (Integer, offset 24, length 4 bytes), if in Schema version Add a new column to this table (Long integer, length 8 bytes, default value is zero) and placed in After that, , , exist The offset relative to None of them changed. exist The offset below is 28, and the difference mapping set record corresponding to this change is the newly added column. Its default value is zero, and its offset is 28. If in subsequent Schema versions... Lieutenant General The length is adjusted from 20 bytes to 30 bytes. All subsequent columns are shifted backward by an offset increment of ten. The difference mapping set records the change in the length of the item type and the corresponding offset increment.

[0031] The calculated difference map set The log is written incrementally to the SSTable layer, a separately configured ghost column in the log structure merge tree. During the write process, no existing cold data physical files are read or modified. The global schema version list is then updated to reflect the new version. Compared to its previous version Connected by pointers, they form a version chain that can be traced bidirectionally along the timeline, allowing subsequent query stages to backtrack as needed to map all differences between any two historical versions.

[0032] When the number or size of the accumulated metadata files in the Ghost Column SSTable layer reaches the preset merge and compression threshold, a merge and compression operation is triggered, merging multiple Ghost Column SSTable files into a more compact metadata file in version order. This merge and compression operation only applies to the Ghost Column metadata layer and does not involve or touch any cold data physical files, thus ensuring that no matter how many times the table structure changes, the cold data physical files always remain in their original immutable state, and only incremental metadata with a size much smaller than the cold data itself needs to be maintained.

[0033] II. Implementation methods for virtualized overwrite retrieval:

[0034] The compute node receives cross-version query requests initiated by users, and the query optimizer parses the query time range, target partition, and target schema version expected to be used in this query. The target schema version is typically the latest effective schema version. The query optimizer determines the historical data blocks involved in this query and their corresponding schema versions at the time of archiving based on the partition archiving information recorded by the metadata management layer. Different historical data blocks correspond to They may be different from each other;

[0035] For each historical data block, determine its corresponding Version matching the target version in this query Consistency: If consistent, the data block is scanned directly according to its native schema without further projection processing. If inconsistent, the process follows the established schema version chain, starting from... Back to Read the difference mapping set corresponding to each version along the way in sequence. The versions are then combined and stacked in chronological order to generate the unified mapping function required for this query. The unified mapping function comprehensively represents all column offset adjustments, type conversions, and default value filling rules from the historical schema to the target schema;

[0036] Continuing with the previous example, if a historical data block is archived in the schema version... The target version for this query is Then the unified mapping function is derived from to Difference mapping (new column) (and its default value) and to Difference mapping (columns) The data is obtained by sequentially stacking and merging (length adjustment and subsequent column offset shift), ensuring that the physical data in this historical data block can be correctly interpreted. Column layout under different versions;

[0037] The schema mapping adapter, embedded before the scan operator of the vectorized execution engine, reads the physical column data of historical data blocks and then applies it according to the unified mapping function. Offset repositioning is performed on each column, type conversion is performed on columns whose type has changed, default values ​​are filled into columns newly added to the target schema but not in the historical data, and deleted historical columns in the target schema are masked, thereby dynamically generating data in memory that conforms to the specified structure. The virtual projection rows of the structure are generated and passed to at least one execution operator in the subsequent filtering, aggregation and join processing in a vectorized batch. The execution engine completes the subsequent calculation processing consistent with the hot data query, so that the query side does not need to be aware of the difference in the original schema version of the underlying data. After the query is completed, the temporary virtual projection buffer generated during the query is released. This buffer only exists during the memory operation. The cold data physical archive file always maintains its original immutable state throughout the entire query process.

[0038] III. Implementation methods for asymmetric hierarchical index structures:

[0039] Before data is archived and written to object storage, a routing layer index item is built for each data block to be archived. The routing layer index item contains statistical information on the value range of each column in the data block, member judgment summary information for quickly determining whether the target key may exist, and the schema version timestamp corresponding to the data block when it is archived. All routing layer index items are loaded into the memory or distributed cache of the computing node to form a global routing index covering all historical data blocks for fast access during the query phase.

[0040] Simultaneously, a retrieval layer index is constructed for each data block to be archived. The retrieval layer index uses either a hash index or a tree index to precisely address at least one fine-grained physical location at the row and column levels within the data block. The retrieval layer index and the cold data physical file are stored in the same logical data block in the object storage.

[0041] When a query request arrives, the routing layer in hot memory first calculates a comprehensive hit score for each candidate data block. The formula for the comprehensive hit score is:

[0042] ;

[0043] In the above formula, Represents data block Total field goal score, Represents data block This is an indicator function that checks whether the statistical range of values ​​overlaps with the range specified in the query conditions. It takes a value of one if there is overlap, and a value of zero if there is no overlap. Indicates based on data block The member-determining summary information is an indicator function that determines whether the target key of the query might exist in the data block. It takes a value of 1 if it might exist, and a value of 0 if it is certain that it does not exist. Represents data block This is an indicator function that checks whether the corresponding schema version timestamp falls within the time window specified in this query; it takes a value of one if it does, and a value of zero if it does not. , , These are preset weight coefficients corresponding to the three indicator functions, used to comprehensively evaluate whether a subsequent network access request needs to be initiated for the data block. The weight coefficients can be adjusted based on historical query statistics.

[0044] Overall hit score For candidate data blocks that reach the preset threshold, only one read request is initiated to read the corresponding retrieval layer index and data content of the data block at the same time, avoiding multiple round-trip network accesses for the same data block. The retrieved retrieval layer index content is precisely hashed or tree-structured locally to locate the precise physical offset position of the target row within the data block. The located physical data is combined with the aforementioned virtualization overwrite retrieval mechanism to perform schema projection transformation, and the final result that conforms to the schema structure of the query target is obtained and returned to the upper-layer execution engine.

[0045] IV. Overall process closed loop:

[0046] The data archiving phase includes the following processing: After hot data meets the archiving conditions, the data is converted into an immutable columnar storage format and written to object storage. The routing layer index and retrieval layer index are built simultaneously. If the data definition language changes during the archiving period, the aforementioned ghost column SSTable generation process is triggered in parallel. This branch does not affect or touch the archived physical files. The data query phase includes the following processing: After receiving a query request, the routing layer quickly filters candidate data blocks, initiates a read request for the candidate data blocks to obtain the retrieval layer index and data, and then enters the virtualization overwrite retrieval process to perform schema projection on the read historical data. Finally, the query results under the unified view are output. The above two branches are interconnected with the ghost column SSTable through a shared schema version linked list, reflecting the closed-loop dependency between the incremental metadata generated in the archiving phase and the projection processing in the query phase.

[0047] V. Other implementation methods:

[0048] In another embodiment of the present invention, instead of uniformly assigning a schema version number to the entire table, a separate version evolution chain is maintained for each column. The difference mapping of each column only records the offset and type information of the column itself in each change. During the query, the schema mapping adapter backtracks its independent version chain for each column involved in the target schema and performs projection transformation, rather than uniformly backtracking the version chain of the entire table. In scenarios where table structure changes are frequent but only concentrated in a few columns, this implementation can further reduce the maintenance granularity and storage overhead of the difference mapping set, and can also achieve a unified query view without rewriting the cold data physical file.

[0049] In another embodiment of the present invention, based on the virtualization overwrite retrieval mechanism, for the combination of historical data blocks and target schemas with high access frequency, the projection results can be cached in read-only mode in the local temporary storage of the computing node. Subsequent repeated queries on the same historical data blocks and the same target schema can directly reuse the cached projection results without having to re-execute column offset relocation and type conversion calculations for each query. This implementation method can further reduce the computational overhead in high-frequency repeated query scenarios while occupying a small amount of additional temporary storage space, and can also achieve the technical objective of the present invention without modifying the underlying cold data physical files.

[0050] The archiving processing method described in this invention can be implemented by a processor in a computing node executing computer program instructions stored in a computer-readable storage medium. The computer-readable storage medium includes, but is not limited to, storage resources provided by a local disk storage, a distributed file system, or an object storage system. When the computer program instructions are read and executed by the processor, the aforementioned processing steps of generating differential metadata layer, virtualization overwrite retrieval, and asymmetric hierarchical index retrieval are implemented.

[0051] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. An archiving method of a distributed database, characterized by, include: When a change in the data definition language of a business table in a distributed database is detected, a new table structure version number is assigned to this change, and the difference mapping set corresponding to this change is calculated. The difference mapping set is written incrementally to the difference metadata layer, which is independent of the cold data physical file. At the same time, the table structure version linked list is updated to record the sequential relationship between adjacent versions. The cold data physical file is an immutable columnar storage format file that has been archived to object storage. The process of writing to the difference metadata layer does not read or modify the cold data physical file. Receive a query request for the cold data physical file, determine the historical table structure version corresponding to the historical data block involved in this query and the target table structure version expected to be used in this query; When the historical table structure version is inconsistent with the target table structure version, the table structure version is traced back from the historical table structure version to the target table structure version, and the difference mapping sets corresponding to each version along the way are read in sequence and superimposed and merged according to the version order to obtain a unified mapping function. Based on the unified mapping function, at least one of the following processes is performed on the physical column data in the historical data block in memory: offset relocation, type conversion, and default value filling, to generate a virtual projection row that conforms to the target table structure version, and the process of generating the virtual projection row does not modify the cold data physical file; A routing layer index is maintained in the memory of the compute node. The routing layer index contains statistical information on the value range of each data block, member judgment summary information, and table structure version timestamp. All data blocks are filtered according to the routing layer index to determine candidate data blocks. A read request is initiated for the determined candidate data blocks to obtain the retrieval layer index and data content corresponding to the data block. The retrieval layer index and the data content are organized in the same logical data block in the object storage and can be obtained as a whole in a single read request. There is no need to initiate separate network accesses for the index and data. The retrieval layer index and the cold data physical file are stored in the same object storage. The system performs a retrieval of the data content locally based on the obtained retrieval layer index, locates the physical offset position of the target row, and performs table structure projection transformation by combining the located physical data with the generation method of the virtual projection row to obtain the query result.

2. The archiving method of a distributed database according to claim 1, characterized in that: The difference mapping set includes the default values ​​of newly added columns, the marking information of deleted columns, field type conversion rules, and the offset increment of each column in the new version relative to the historical version.

3. The archiving method of a distributed database according to claim 1, characterized in that: The differential metadata layer is a metadata layer set up independently in the log structure merging tree. When the number or size of the accumulated metadata files in the differential metadata layer reaches a preset threshold, a merging and compression process is performed on the differential metadata layer. The merging and compression process does not involve the cold data physical files.

4. The archiving method of a distributed database according to claim 1, characterized in that: The steps to determine the historical table structure version corresponding to the historical data blocks involved in this query include: obtaining the table structure version corresponding to each historical data block when it was archived based on the partition archiving information recorded by the metadata management layer; and when the historical table structure version is consistent with the target table structure version, directly scanning the corresponding historical data blocks according to the original table structure.

5. The archiving method of a distributed database according to claim 1, characterized in that: The step of generating the virtual projection rows is completed in a table structure mapping adapter embedded before the scan operator of the vectorized execution engine. The virtual projection rows are passed in vectorized batches to at least one execution operator in subsequent filtering, aggregation, and join processes.

6. The archiving method of a distributed database according to claim 1, wherein: After the query is completed, the temporary projection buffer generated during the process of generating the virtual projection row is released, and the cold data physical file remains in its original immutable state throughout the query process.

7. The archiving method for a distributed database according to claim 1, characterized in that: The steps for determining candidate data blocks based on the routing layer index include: for each candidate data block, determining whether the query target may exist in the data block based on whether the statistical information of the value range of the data block overlaps with the value range specified in this query, the member determination summary information of the data block, and whether the table structure version timestamp corresponding to the data block falls within the time window specified in this query, calculating the comprehensive hit score of the data block, and determining the data block whose comprehensive hit score reaches a preset threshold as a candidate data block.

8. The archiving method for a distributed database according to claim 1, characterized in that: The retrieval layer index is one of a hash index and a tree index, used to address at least one fine-grained physical location at the row level and column level within a data block.

9. The archiving method for a distributed database according to claim 1, characterized in that: The offset increment of each column in the difference mapping set is calculated by summing the offset increments of all previous columns introduced in this change.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the archive processing method for a distributed database as described in any one of claims 1 to 9.