Database index bloat cleanup method and apparatus, computer program product

Abstract

This invention proposes a database index bloat cleanup method, apparatus, and computer program product. The method includes: traversing physical index keys, segmenting the physical index keys to generate multiple slice tasks; distributing the multiple slice tasks to multiple concurrent execution units for parallel processing; each concurrent execution unit sequentially concatenating and restoring each target physical index key within the logical boundary information range, and performing the following operations for each target physical index key: reading the index value information corresponding to the target physical index key and decoding it to obtain a set of data tuple reference information; traversing each data tuple reference information in the data tuple reference information set and performing a verification operation; when it is determined that all data tuple reference information corresponding to the target physical index key is invisible, determining that the target physical index key is a key to be deleted; and passing the key to be deleted to a preset asynchronous processing channel to perform a deletion operation. This invention can shorten the overall execution time of index bloat cleanup.

Description

Technical Field

[0001] This invention relates to the field of database management and storage technology, and in particular to a database index bloat cleanup method and apparatus, and a computer program product. Background Technology

[0002] With the rapid development of cloud computing and distributed storage technologies, high-performance distributed relational databases built on underlying key-value (KV) storage systems (such as FoundationDB) are becoming increasingly widely used. In this storage-compute separation or layered architecture, to accelerate the retrieval efficiency of massive amounts of data, database systems typically create a large number of secondary indexes or primary key indexes. Meanwhile, to ensure data consistency and read / write performance under high concurrency scenarios, mainstream databases generally adopt a multi-version concurrency control (MVCC) mechanism. This means that when data is updated or deleted, the old data is not directly overwritten or immediately physically cleared, but rather recorded by appending a new version. However, the long-term operation of the MVCC mechanism inevitably leads to a serious data bloat problem. Especially for index structures, with the commit and rollback of a large number of historical transactions, a massive amount of expired, invisible, and invalid data tuple references remain in the underlying storage system. If these bloated garbage index data are not cleaned up in time, they will not only consume a large amount of valuable storage space but also significantly increase the scan range during queries, leading to a severe decline in the overall read / write performance of the database.

[0003] In existing technologies, cleaning up database index bloat typically relies on traditional single-threaded row-by-row scanning or garbage collection mechanisms driven by the master table. However, these conventional methods expose many limitations when dealing with massive distributed data. Existing index scanning mechanisms usually read data using fixed physical offsets or full serial traversal. When faced with massive and unevenly distributed underlying physical index keys, they lack efficient and smooth logical partitioning methods. Furthermore, the locking granularity of existing index cleanup operations is often coarse (e.g., only supporting table-level locks). When processing large tables, the locking mechanism limits the efficiency of execution by increasing concurrency, often leading to excessive single-threaded load, easy memory overflow, or long cleanup cycles due to insufficient concurrency. In addition, traditional scan requests typically retrieve complete key-value pair data at once, causing significant network bandwidth consumption and resource waste during the pre-screening stage.

[0004] In the process of identifying and verifying the validity of indexes, existing technologies incur significant overhead for cross-layer visibility verification, resulting in a severe "read amplification" effect. Because index information and actual data tuples are typically stored decoupled and separately in the underlying key-value storage model, existing technologies often require frequently fetching entire pages or complete data tuples from the underlying layer to the computation layer for state resolution when determining whether an index record should be deleted. This coarse-grained verification method, lacking precise point-to-point tracing and independent snapshot comparison, is highly susceptible to lock conflicts in complex, concurrent transaction environments, thereby blocking other normal business operations that require data modification while ensuring data consistency.

[0005] In the final cleanup execution phase, existing technologies often trigger I / O storms in the underlying storage engine and exacerbate performance degradation. After identifying the garbage index keys to be deleted, existing mechanisms generally employ a synchronous, single-key, one-by-one call approach to perform physical deletion operations. In distributed systems, such frequent single-point remote procedure call requests not only lead to high network communication latency but also trigger a large number of random network and disk interactions during large-scale garbage cleanup, severely consuming underlying network and I / O resources and thus seriously interfering with the database's response and throughput capabilities for normal business requests. Summary of the Invention

[0006] The purpose of this invention is to provide a database index bloat cleanup method and apparatus, and a computer program product, to solve the above-mentioned problems, which can significantly improve the overall execution concurrency of the cleanup operation and greatly shorten the overall execution time of index bloat cleanup.

[0007] This invention proposes a method for cleaning up database index bloat, comprising the following steps: The physical index keys in the underlying key-value storage system are traversed and scanned, and the physical index keys are divided according to the preset batch size to generate multiple slice tasks containing logical boundary information. The multiple slice tasks are distributed to multiple concurrent execution units for parallel processing; Each concurrent execution unit, based on the logical boundary information received from the sliced ​​tasks, sequentially reconstructs each target physical index key within the range of the logical boundary information, and performs the following operations for each target physical index key: The index value information corresponding to the target physical index key is read from the underlying key-value storage system, and the data tuple reference information set is decoded. Traverse each data tuple reference information in the data tuple reference information set and perform a verification operation: generate a data tuple key by concatenating the data tuple row identifier in the currently traversed data tuple reference information, read the corresponding data tuple wrapping information from the underlying key-value storage system, and extract the corresponding target version information from the data tuple wrapping information based on the version identifier in the data tuple reference information. Then, determine whether the data tuple reference information is visible based on the target version information. When it is determined that all data tuple reference information corresponding to the target physical index key is not visible, the target physical index key is determined to be a key to be deleted; The key to be deleted is passed to a preset asynchronous processing channel, and the deletion operation of the key to be deleted is executed by calling the storage engine interface.

[0008] In one embodiment, the step of traversing and scanning the physical index keys in the underlying key-value storage system, and splitting the physical index keys according to a preset batch size to generate multiple slice tasks containing logical boundary information, specifically includes: A streaming read request is initiated by calling the range query interface of the underlying key-value storage system to retrieve only the physical index key. The streaming read request carries a preset batch size as the upper limit parameter for the number of reads in a single session. Based on the current returned result, extract the first and last physical index keys to generate a slice task containing logical boundary information; The last physical index key is used as the starting key cursor for the next batch of streaming read requests. The range query interface call is executed repeatedly until all physical index keys in the underlying key-value storage system are traversed to generate multiple slice tasks.

[0009] In one embodiment, the step of sequentially concatenating and restoring the target physical index keys within the logical boundary information range specifically includes: Obtain the namespace prefix, logical partition identifier, and relational path of the target index table pre-configured in the current database system; The encoded data of each index column within the range defined by the logical boundary information are determined sequentially; The namespace prefix, logical partition identifier, relation path, and index column encoded data are concatenated according to a preset hierarchical order and delimiter to generate the target physical index key.

[0010] In one embodiment, determining whether the data tuple reference information is visible based on the target version information specifically includes: Extract the creation transaction identifier and deletion transaction identifier recorded in the target version information; Obtain a snapshot of active transactions in the current database system, the snapshot containing a list of currently running and uncommitted transaction identifiers; If the transaction indicated by the created transaction identifier has been committed and is not in the transaction snapshot, and the deleted transaction identifier is empty, invalid, or the transaction it indicates is included in the transaction snapshot, then the data tuple reference information is determined to be visible. If the transaction indicated by the creation transaction identifier has not been committed, has been rolled back, or is included in the transaction snapshot, or if the transaction indicated by the deletion transaction identifier has been committed but is not in the transaction snapshot, then the data tuple reference information is determined to be invisible.

[0011] In one embodiment, for each target physical index key, the method further includes: When it is determined that only a portion of the data tuple reference information in the data tuple reference information set corresponding to the target physical index key is not visible, the invisible data tuple reference information is removed from the data tuple reference information set. The remaining visible data tuple reference information is re-encoded to generate new index value information; The new index value information is written back to the underlying key-value storage system to overwrite the index value information corresponding to the target physical index key.

[0012] In one embodiment, before passing the key to be deleted to a preset asynchronous processing channel, the method further includes: After each target physical index key in the current slice task has been processed, determine whether the keys marked as keys to be deleted among the multiple target physical index keys are consecutive in physical storage according to the order of the index keys; If the keys are consecutive, the consecutive keys to be deleted will be merged into a range, and the current merging will end when the adjacent keys to be deleted are no longer consecutive, generating the range information to be deleted. Accordingly, the key to be deleted is passed to a preset asynchronous processing channel, and the deletion operation of the key to be deleted is performed by calling the storage engine interface, including: passing the information of the range to be deleted to the asynchronous processing channel, so as to perform a batch deletion operation by calling the storage engine interface.

[0013] In one embodiment, the asynchronous processing channel is configured with multiple concurrent execution units; The step of performing the deletion operation on the key to be deleted by calling the storage engine interface specifically includes: The multiple concurrent execution units concurrently extract the key or range information to be deleted from the asynchronous processing channel, and complete the physical deletion operation by calling the application programming interface of the underlying key-value storage system; After all concurrent execution units have completed scanning of their corresponding slice tasks and all keys or ranges to be deleted have been physically deleted, the expansion cleanup of the target index table is considered complete.

[0014] This invention also proposes a database index bloat cleanup device for implementing the database index bloat cleanup method described above, comprising: The task segmentation module is used to traverse and scan the physical index keys in the underlying key-value storage system, and segment the physical index keys according to the preset batch size to generate multiple slice tasks containing logical boundary information. The task distribution module is used to distribute the multiple slice tasks to a parallel processing cluster consisting of multiple concurrent execution units for parallel processing. The key restoration module is used to sequentially concatenate and restore the physical index keys of each target within the logical boundary information range based on the logical boundary information in the received slice task when the concurrent execution unit processes the data in parallel. The traceability and verification module is used to, when processed in parallel by the concurrent execution unit, read the index value information corresponding to the target physical index key from the underlying key-value storage system for each target physical index key, decode it to obtain a set of data tuple reference information, and traverse each data tuple reference information in the set of data tuple reference information to perform a verification operation: generate a data tuple key by concatenating the data tuple row identifier in the currently traversed data tuple reference information, read the corresponding data tuple wrapping information from the underlying key-value storage system, and extract the corresponding target version information from the data tuple wrapping information based on the version identifier in the data tuple reference information, and determine whether the data tuple reference information is visible based on the target version information; The status marking module is used to determine that the target physical index key is a key to be deleted when all data tuple reference information corresponding to the target physical index key is not visible during parallel processing by the concurrent execution unit. The asynchronous deletion module is used to pass the key to be deleted to a preset asynchronous processing channel and execute the deletion operation of the key to be deleted by calling the storage engine interface.

[0015] In one embodiment, the database index bloat cleanup device further includes a partial recycling module. The partial recycling module is used to, when the concurrent execution unit processes the data tuple reference information in parallel, remove the invisible data tuple reference information from the data tuple reference information set corresponding to the target physical index key when it is determined that only a portion of the data tuple reference information is not visible, re-encode the remaining visible data tuple reference information to generate new index value information, and write the new index value information back to the underlying key-value storage system to overwrite the index value information corresponding to the target physical index key.

[0016] In one embodiment, the asynchronous deletion module is further configured to, after processing each target physical index key in the current slice task, determine whether multiple target physical index keys marked as keys to be deleted are consecutive in physical storage according to the order of the index keys. If they are consecutive, the consecutive keys to be deleted are merged in range, and the current merging ends when adjacent keys to be deleted are not consecutive, generating information to be deleted in the range, and transmitting the information to be deleted in the range to the asynchronous processing channel to perform batch deletion operations by calling the storage engine interface.

[0017] The present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the database index bloat cleanup method described above.

[0018] Compared with the prior art, the beneficial effects of the database index bloat cleaning method and apparatus, and computer program product of the present invention are as follows: 1) This invention refines the scanning and processing granularity of index bloat cleanup from the traditional table level to the specific physical index key level. By traversing and scanning the physical index keys in the underlying key-value storage system, and generating slice tasks containing logical boundary information according to a preset batch size, the tasks are distributed to multiple concurrent execution units for parallel processing, effectively overcoming the load bottleneck caused by traditional single-threaded scanning. This method can smoothly and evenly slice massive and unevenly distributed index data, significantly improving the overall execution concurrency of scanning and cleanup operations, greatly shortening the overall execution time of index bloat cleanup, thereby effectively reducing the impact of cleanup operations on normal business operations. At the same time, in the pre-screening stage, only the key fields are read without fetching complete key-value pairs, which greatly reduces network bandwidth consumption and memory occupation of computing nodes, effectively avoiding the risk of memory overflow when processing large tables.

[0019] 2) This invention reads and decodes index value information from the underlying key-value storage system, combines data tuple row identifiers and version identifiers to accurately trace the corresponding data tuple wrapper information to extract the target version information, and performs visibility verification based on creation transaction identifiers, deletion transaction identifiers, and active transaction snapshots. This achieves fine-grained independent state resolution, avoiding the read amplification problem caused by frequently pulling entire pages or complete data tuples to the computation layer. This lightweight verification logic, while ensuring data consistency in high-concurrency transaction environments, reduces lock conflicts caused by coarse-grained locking mechanisms and reduces the blocking impact of cleanup operations on normal business modification operations.

[0020] 3) This invention utilizes a pre-defined asynchronous processing channel and multiple configured concurrent execution units to perform physical deletion operations. After the slice task is completed, based on the continuity of physical storage, keys with consecutive identical operations to be deleted are compressed and ranges merged to generate deletion interval information. Then, batch cleanup is performed by calling the storage engine interface. This not only effectively isolates the time-consuming physical deletion task from the normal main business process but also transforms a large number of single-point random remote procedure calls into efficient batch operations, significantly reducing network communication latency and the input / output overhead of the underlying storage engine. When facing large system index tables with extremely high garbage expansion rates, this batch processing and merging mechanism can significantly shorten cleanup time, resulting in a substantial increase in execution efficiency. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a database index bloat cleanup method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a database index bloat cleanup device according to an embodiment of the present invention.

[0022] Figure Labels

[0023] 100. Task Splitting Module; 200. Task Distribution Module; 300. Key Restoration Module; 400. Source Tracing and Verification Module; 500. Status Marking Module; 600. Asynchronous Deletion Module; 700. Partial Recycling Module. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention more readily understood, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that while many specific details are set forth in the following description to provide a thorough understanding of the invention, the invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of the invention; therefore, the invention is not limited to the specific embodiments disclosed below.

[0025] Secondly, the collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0026] Secondly, the phrase "an embodiment" or "an embodiment" in this application refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the invention. The phrases "in one embodiment" and "an embodiment" appearing in different places in this specification do not all refer to the same embodiment, nor are they embodiments that are mutually exclusive, either alone or selectively, with other embodiments. The terms "comprising" or "including" indicate the presence of the claimed feature but do not exclude the presence of one or more other features. The term "and / or" as used in this application includes any and all combinations of one or more of the related listed items.

[0027] This invention proposes a method for cleaning up database index bloat, see [link to relevant documentation]. Figure 1 It includes the following steps: Step S1: Traverse and scan the physical index keys in the underlying key-value storage system, and divide the physical index keys according to the preset batch size to generate multiple slice tasks containing logical boundary information. Step S2: Distribute the multiple slice tasks to multiple concurrent execution units for parallel processing; Step S3: Each concurrent execution unit, based on the logical boundary information received from the sliced ​​task, sequentially reconstructs each target physical index key within the range of the logical boundary information, and performs the following operations for each target physical index key: Step S4: Read the index value information corresponding to the target physical index key from the underlying key-value storage system, and decode it to obtain a set of data tuple reference information; Step S5: Traverse each data tuple reference information in the data tuple reference information set and perform a verification operation: Generate a data tuple key by concatenating the data tuple row identifier in the currently traversed data tuple reference information, read the corresponding data tuple wrapping information from the underlying key-value storage system, and extract the corresponding target version information from the data tuple wrapping information based on the version identifier in the data tuple reference information. Then, determine whether the data tuple reference information is visible based on the target version information. Step S6: When it is determined that all data tuple reference information corresponding to the target physical index key is not visible, the target physical index key is determined to be a key to be deleted. Step S7: Pass the key to be deleted into a preset asynchronous processing channel, and execute the deletion operation of the key to be deleted by calling the storage engine interface.

[0028] Specifically, in large-scale relational distributed databases, FoundationDB can be used as the underlying key-value store. Due to the layout design of the index tuple keys, keys in FoundationDB are not evenly distributed, making it impossible to use the heap tuple partitioning method. Therefore, it is necessary to first traverse and scan the index keys, partition the data according to a preset batch size, and then distribute the partitioned tasks to N concurrent execution units (such as N coroutines) for concurrent execution. Each partitioned task will perform subsequent concatenation, reading, and verification operations based on the passed partition boundary information (start row, end row).

[0029] In a database index bloat cleanup method according to one embodiment of the present invention, the step of traversing and scanning the physical index keys in the underlying key-value storage system and splitting the physical index keys according to a preset batch size to generate multiple slice tasks containing logical boundary information specifically includes: initiating a streaming read request to obtain only physical index keys by calling the range query interface of the underlying key-value storage system, wherein the streaming read request carries a preset batch size as an upper limit parameter for the number of reads in a single session; extracting the first and last physical index keys from the current returned results to generate slice tasks containing logical boundary information; using the last physical index key as the starting key cursor for the next batch of streaming read requests, and repeatedly executing the range query interface call until all physical index keys in the underlying key-value storage system have been traversed to generate multiple slice tasks. As a specific underlying interaction logic, when performing the pre-traversal scan, the FoundationDB streaming range read API is called, and the key-only mode is specified, ignoring the reading of large amounts of value data. Once a batch size of keys has been scanned, the last key is recorded as a cursor, and the boundary information for that batch is generated. This mechanism ensures that the master node can safely partition hundreds of millions of index data with very low memory footprint.

[0030] In a database index bloat cleanup method according to an embodiment of the present invention, the step of sequentially concatenating and restoring each target physical index key within the logical boundary information range specifically includes: obtaining the namespace prefix, logical partition identifier, and relational path of the target index table pre-configured in the current database system; sequentially determining the encoded data of each index column within the logical boundary information range; and concatenating the namespace prefix, logical partition identifier, relational path, and the encoded data of the index columns according to a preset hierarchical order and delimiter to generate the target physical index key. In the underlying metadata service, index tuple data exists in the FoundationDB storage space in the form of key-value pairs. Its key layout design can be represented as: 'prefix:cs:{logicpart}:idx / {relpath} / {indexkey}', where prefix corresponds to the namespace prefix, logicpart corresponds to the logical partition identifier, relpath corresponds to the relational path, and indexkey is the specific data encoding of the index column. Furthermore, the value part is encoded and packaged in the form of an ObjValues ​​structure, mainly using the Refs field. This field is typically defined as an object value reference structure containing a data tuple row identifier (Rownum) and an index number (Idx), representing a specific version of a heap tuple. Therefore, an index tuple can be simply understood as recording all heap tuple versions corresponding to a given index key. The coroutine constructs the heap tuple key field (each key corresponding to a row in the heap tuple table) based on its Rowum, reads the corresponding value information, and finally retrieves the corresponding version information using the Idx from the reference information.

[0031] In a database index bloat cleanup method according to an embodiment of the present invention, the step of determining whether the data tuple reference information is visible based on the target version information specifically includes: extracting the creation transaction identifier and deletion transaction identifier recorded in the target version information; obtaining a snapshot of transactions currently active in the database system, the transaction snapshot containing a list of currently running and uncommitted transaction identifiers; if the transaction indicated by the creation transaction identifier has been committed and is not in the transaction snapshot, and the deletion transaction identifier is empty, invalid, or the transaction it indicates is included in the transaction snapshot, then the data tuple reference information is determined to be visible; if the transaction indicated by the creation transaction identifier has not been committed, has been rolled back, or is included in the transaction snapshot, or the transaction indicated by the deletion transaction identifier has been committed and is not in the transaction snapshot, then the data tuple reference information is determined to be invisible. The visibility determination relies on the Multi-Version Concurrency Control (MVCC) mechanism. The system extracts metadata from the header of the heap tuple version (such as the insert transaction ID t_xmin, which serves as the identifier for the creation transaction, and the delete or update transaction ID t_xmax, which serves as the identifier for the deletion transaction), and compares it with the current database's active transaction snapshot. If the creation transaction identifier (such as t_xmin) has been committed and is not visible in the current snapshot, or the deletion transaction identifier (such as t_xmax) is valid and visible, then the system accurately determines that the version is invisible (dead tuple) and marks it accordingly.

[0032] In a database index bloat cleanup method according to an embodiment of the present invention, for each target physical index key, the method further includes: when it is determined that only some data tuple reference information in the data tuple reference information set corresponding to the target physical index key is invisible, removing the invisible data tuple reference information from the data tuple reference information set; re-encoding the remaining visible data tuple reference information to generate new index value information; and writing the new index value information back to the underlying key-value storage system to overwrite the index value information corresponding to the target physical index key. Specifically, after completing the visibility determination of all versions under a single index key, when all heap tuple reference information corresponding to the index has been traversed and determined, if it is found that only some references in the reference set are invisible (i.e., the whole row deletion standard of "all references are invisible" has not been met), the coroutine will remove the invisible heap tuple reference objects from the original set, re-encode and package the remaining valid heap tuple reference objects to generate new value information (ObjValues ​​structure), and write it back to FoundationDB via API to overwrite the original value, thereby achieving accurate recycling of some garbage data within the row. If all references corresponding to the index are no longer visible (have been deleted), then the line is marked as pending deletion.

[0033] In a database index bloat cleanup method according to an embodiment of the present invention, before passing the key to be deleted to a preset asynchronous processing channel, the method further includes: after processing each target physical index key in the current slice task, determining whether the keys marked as keys to be deleted among the multiple target physical index keys are consecutive in physical storage according to the order of the index keys; if they are consecutive, merging the consecutive keys to be deleted into ranges, and ending the current merging when adjacent keys to be deleted are not consecutive, generating information on the interval to be deleted; correspondingly, passing the key to be deleted to the preset asynchronous processing channel, and performing the deletion operation of the key to be deleted by calling the storage engine interface, including: passing the information on the interval to be deleted to the asynchronous processing channel to perform a batch deletion operation by calling the storage engine interface.

[0034] In a database index bloat cleanup method according to an embodiment of the present invention, the asynchronous processing channel is configured with multiple concurrent execution units. The step of executing the deletion operation of the key to be deleted by calling the storage engine interface specifically includes: the multiple concurrent execution units concurrently extracting the key to be deleted or the range to be deleted information from the asynchronous processing channel, and completing the physical deletion operation by calling the application programming interface of the underlying key-value storage system; after all concurrent execution units have completed scanning the corresponding slice tasks and all the key to be deleted or the range to be deleted information has completed the physical deletion operation, the bloat cleanup of the target index table is determined to be completed. Specifically, after all index keys in the slice have been executed, the consecutive rows to be deleted within the slice are merged, and finally, the key range information to be deleted is passed into the deletion channel. Subsequently, the system starts N coroutines to continuously retrieve key range information from the deletion channel and batch delete the corresponding rows or row ranges through the FoundationDB API. In actual deployment testing, taking the system index table pg_class_oid_index as an example, the number of valid tuples was 9,836,510, and the number of "dead tuples" was 463,086,586, resulting in a garbage bloat rate of 97.9%. Using Greenplum's original index bloat cleanup method, the process took 7 hours and 29 minutes; using the cleanup method of this invention, the time was reduced to within 2 hours, improving efficiency by 374%.

[0035] This invention also proposes a database index bloat cleanup device, see [link to relevant documentation]. Figure 2This system is used to implement the database index bloat cleanup method described above, and includes a task splitting module, a task distribution module, a key restoration module, a source verification module, a status marking module, an asynchronous deletion module, and a partial recycling module. The task splitting module iterates through the physical index keys in the underlying key-value storage system and splits the physical index keys according to a preset batch size, generating multiple slice tasks containing logical boundary information. The task distribution module distributes the multiple slice tasks to a parallel processing cluster composed of multiple concurrent execution units for parallel processing. The key restoration module, during parallel processing by the concurrent execution units, sequentially reconstructs each target physical index key within the range of the received logical boundary information based on the logical boundary information in the slice tasks. The source verification module, during parallel processing by the concurrent execution unit, reads the index value information corresponding to each target physical index key from the underlying key-value storage system, decodes it to obtain a set of data tuple reference information, and iterates through each data tuple reference information in the set to perform a verification operation: it generates a data tuple key by concatenating the data tuple row identifiers in the currently traversed data tuple reference information, reads the corresponding data tuple wrapping information from the underlying key-value storage system, extracts the corresponding target version information from the data tuple wrapping information based on the version identifier in the data tuple reference information, and determines whether the data tuple reference information is visible based on the target version information. The status marking module, during parallel processing by the concurrent execution unit, determines that the target physical index key is a key to be deleted when it is determined that all data tuple reference information corresponding to the target physical index key is invisible. The asynchronous deletion module passes the key to be deleted to a preset asynchronous processing channel and executes the deletion operation of the key to be deleted by calling the storage engine interface. The partial recycling module is used to, when the concurrent execution unit processes data in parallel, remove the invisible data tuple reference information from the data tuple reference information set corresponding to the target physical index key when it is determined that only a portion of the data tuple reference information is not visible, re-encode the remaining visible data tuple reference information to generate new index value information, and write the new index value information back to the underlying key-value storage system to overwrite the index value information corresponding to the target physical index key.

[0036] In one embodiment of the present invention, the asynchronous deletion module is further configured to, after the processing of each target physical index key in the current slice task is completed, determine whether the multiple target physical index keys marked as keys to be deleted are consecutive in physical storage according to the order of the index keys. If they are consecutive, the consecutive keys to be deleted are merged in range, and the current merging ends when adjacent keys to be deleted are not consecutive, generating information to be deleted intervals. The information to be deleted intervals is then passed to the asynchronous processing channel to perform batch deletion operations by calling the storage engine interface.

[0037] In actual devices, the above modules can be instantiated and run by the processor calling the program code stored in memory.

[0038] The present invention also proposes a computer program product, including a computer program that, when executed by a processor, implements the database index bloat cleanup method described above.

[0039] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0040] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0041] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0042] The constructions and arrangements of this application shown in several different exemplary embodiments are merely illustrative. Although only a few embodiments are described in detail in this disclosure, those who consult this disclosure will readily understand that many modifications are possible (e.g., changes in size, structure, shape, and proportions, as well as parameter values, installation arrangements, use of materials, color, orientation, etc.) without substantially departing from the novel teachings and advantages of the subject matter described in this application. For example, an element shown as integrally formed may be composed of multiple parts or elements, the position of the elements may be inverted or otherwise changed, and the nature or number or position of discrete elements may be altered or changed. Therefore, all such modifications are intended to be included within the scope of this invention. The order or sequence of any process or method steps may be changed or rearranged according to alternative embodiments. In the claims, any "device plus function" clause is intended to cover the structure described herein that performs the function, and not only structurally equivalent but also equivalent in structure. Various corresponding modifications and variations can be made by those skilled in the art according to this invention without departing from the spirit and essence of the invention, but such corresponding modifications and variations should fall within the protection scope of this invention.

Claims

1. A method for cleaning up database index bloat, characterized in that, Includes the following steps: The physical index keys in the underlying key-value storage system are traversed and scanned, and the physical index keys are divided according to the preset batch size to generate multiple slice tasks containing logical boundary information. The multiple slice tasks are distributed to multiple concurrent execution units for parallel processing; Each concurrent execution unit, based on the logical boundary information received from the sliced ​​tasks, sequentially reconstructs each target physical index key within the range of the logical boundary information, and performs the following operations for each target physical index key: The index value information corresponding to the target physical index key is read from the underlying key-value storage system, and the data tuple reference information set is decoded. Traverse each data tuple reference information in the data tuple reference information set and perform a verification operation: generate a data tuple key by concatenating the data tuple row identifier in the currently traversed data tuple reference information, read the corresponding data tuple wrapping information from the underlying key-value storage system, and extract the corresponding target version information from the data tuple wrapping information based on the version identifier in the data tuple reference information. Then, determine whether the data tuple reference information is visible based on the target version information. When it is determined that all data tuple reference information corresponding to the target physical index key is not visible, the target physical index key is determined to be a key to be deleted. The key to be deleted is passed to a preset asynchronous processing channel, and the deletion operation of the key to be deleted is performed by calling the storage engine interface.

2. The database index bloat cleanup method according to claim 1, characterized in that, The process of traversing and scanning the physical index keys in the underlying key-value storage system, and splitting the physical index keys according to a preset batch size to generate multiple slice tasks containing logical boundary information, specifically includes: A streaming read request is initiated by calling the range query interface of the underlying key-value storage system to retrieve only the physical index key. The streaming read request carries a preset batch size as the upper limit parameter for the number of reads in a single session. Based on the current returned result, extract the first and last physical index keys to generate a slice task containing logical boundary information; The last physical index key is used as the starting key cursor for the next batch of streaming read requests. The range query interface call is executed repeatedly until all physical index keys in the underlying key-value storage system are traversed to generate multiple slice tasks.

3. The database index bloat cleanup method according to claim 1, characterized in that, The step of sequentially concatenating and restoring the physical index keys of each target within the logical boundary information range specifically includes: Obtain the namespace prefix, logical partition identifier, and relational path of the target index table pre-configured in the current database system; The encoded data of each index column within the range defined by the logical boundary information are determined sequentially; The namespace prefix, logical partition identifier, relation path, and index column encoded data are concatenated according to a preset hierarchical order and delimiter to generate the target physical index key.

4. The database index bloat cleanup method according to claim 1, characterized in that, The step of determining whether the reference information of the data tuple is visible based on the target version information specifically includes: Extract the creation transaction identifier and deletion transaction identifier recorded in the target version information; Obtain a snapshot of active transactions in the current database system, the snapshot containing a list of currently running and uncommitted transaction identifiers; If the transaction indicated by the created transaction identifier has been committed and is not in the transaction snapshot, and the deleted transaction identifier is empty, invalid, or the transaction it indicates is included in the transaction snapshot, then the data tuple reference information is determined to be visible. If the transaction indicated by the creation transaction identifier has not been committed, has been rolled back, or is included in the transaction snapshot, or if the transaction indicated by the deletion transaction identifier has been committed but is not in the transaction snapshot, then the data tuple reference information is determined to be invisible.

5. The database index bloat cleanup method according to claim 4, characterized in that, For each target physical index key, the method further includes: When it is determined that only a portion of the data tuple reference information in the data tuple reference information set corresponding to the target physical index key is not visible, the invisible data tuple reference information is removed from the data tuple reference information set. The remaining visible data tuple reference information is re-encoded to generate new index value information; The new index value information is written back to the underlying key-value storage system to overwrite the index value information corresponding to the target physical index key.

6. The database index bloat cleanup method according to claim 1, characterized in that, Before passing the key to be deleted to the preset asynchronous processing channel, the process also includes: After each target physical index key in the current slice task has been processed, determine whether the keys marked as keys to be deleted among the multiple target physical index keys are consecutive in physical storage according to the order of the index keys; If the keys are consecutive, the consecutive keys to be deleted will be merged into a range, and the current merging will end when the adjacent keys to be deleted are no longer consecutive, generating the range information to be deleted. Accordingly, the key to be deleted is passed to a preset asynchronous processing channel, and the deletion operation of the key to be deleted is performed by calling the storage engine interface, including: passing the information of the range to be deleted to the asynchronous processing channel, so as to perform a batch deletion operation by calling the storage engine interface.

7. The database index bloat cleanup method according to claim 6, characterized in that, The asynchronous processing channel is configured with multiple concurrent execution units; The step of performing the deletion operation on the key to be deleted by calling the storage engine interface specifically includes: The multiple concurrent execution units concurrently extract the key or range information to be deleted from the asynchronous processing channel, and complete the physical deletion operation by calling the application programming interface of the underlying key-value storage system; After all concurrent execution units have completed scanning of their corresponding slice tasks and all keys or ranges to be deleted have been physically deleted, the expansion cleanup of the target index table is considered complete.

8. A database index bloat cleanup device, characterized in that, A method for implementing the database index bloat cleanup method as described in any one of claims 1-7 includes: The task segmentation module is used to traverse and scan the physical index keys in the underlying key-value storage system, and segment the physical index keys according to the preset batch size to generate multiple slice tasks containing logical boundary information. The task distribution module is used to distribute the multiple slice tasks to a parallel processing cluster consisting of multiple concurrent execution units for parallel processing. The key restoration module is used to sequentially concatenate and restore the physical index keys of each target within the logical boundary information range based on the logical boundary information in the received slice task when the concurrent execution unit processes the data in parallel. The traceability and verification module is used to, when processed in parallel by the concurrent execution unit, read the index value information corresponding to the target physical index key from the underlying key-value storage system for each target physical index key, decode it to obtain a set of data tuple reference information, and traverse each data tuple reference information in the set of data tuple reference information to perform a verification operation: generate a data tuple key by concatenating the data tuple row identifier in the currently traversed data tuple reference information, read the corresponding data tuple wrapping information from the underlying key-value storage system, and extract the corresponding target version information from the data tuple wrapping information based on the version identifier in the data tuple reference information, and determine whether the data tuple reference information is visible based on the target version information; The status marking module is used to determine that the target physical index key is a key to be deleted when all data tuple reference information corresponding to the target physical index key is not visible during parallel processing by the concurrent execution unit. The asynchronous deletion module is used to pass the key to be deleted to a preset asynchronous processing channel and execute the deletion operation of the key to be deleted by calling the storage engine interface.

9. The database index bloat cleanup device according to claim 8, characterized in that, It also includes a partial recycling module, which is used to, when the concurrent execution unit processes in parallel, remove the invisible data tuple reference information from the data tuple reference information set corresponding to the target physical index key when it is determined that only a portion of the data tuple reference information is not visible, re-encode the remaining visible data tuple reference information to generate new index value information, and write the new index value information back to the underlying key-value storage system to overwrite the index value information corresponding to the target physical index key.

10. The database index bloat cleanup device according to claim 8, characterized in that, The asynchronous deletion module is also used to determine whether multiple target physical index keys marked as keys to be deleted are consecutive in physical storage after the processing of each target physical index key in the current slice task is completed, according to the order of the index keys. If they are consecutive, the consecutive keys to be deleted are merged in range, and the current merging ends when adjacent keys to be deleted are not consecutive, generating information to be deleted intervals. The information to be deleted intervals is then passed to the asynchronous processing channel to perform batch deletion operations by calling the storage engine interface.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the database index bloat cleanup method as described in any one of claims 1-7.

Images