A column-store engine compact with write transaction non-conflict concurrent processing method

Abstract

The present application relates to the technical field of database and big data storage concurrent control, and discloses a column storage engine Compact and write transaction conflict-free concurrent processing method, comprising the following steps: a Compact transaction filters source files based on a snapshot and reads valid data rows; the valid data rows are reorganized to generate a target data file, and a position mapping relationship from a source row to a target row is constructed; a foreground write transaction locally caches operation intentions in an execution stage, and does not modify a global visibility state; when the Compact transaction is submitted, control authority is acquired, information is migrated and deleted based on the position mapping relationship, the source file is atomically marked as invalid, and a target file visible version is published; when the write transaction is submitted, a source file visibility bitmap is directly written or a target file visibility bitmap is written after being converted based on the position mapping relationship according to a source file state; and the position mapping relationship is asynchronously recycled based on a global transaction water level in the background.

Description

Technical Field

[0001] This invention relates to the field of database and big data storage concurrency control technology, and in particular to a method for conflict-free concurrent processing of columnar storage engine Compact and write transactions. Background Technology

[0002] In compute-store separation architectures, object storage, and cloud-native database applications, columnar storage engines typically decompose data tables into multiple immutable data files for persistent storage. The continuous data writing process generates a large number of small-granularity data files, leading to amplified object storage access, increased file scanning by queries, reduced data compression rates, and escalating metadata management overhead. The industry generally employs a backend compaction and merging mechanism to reorganize, sort, and recompress these small, fragmented files to generate new files, thereby optimizing storage efficiency and query performance. Meanwhile, modern databases use multi-version concurrency control mechanisms to ensure transaction isolation. Columnar storage engines rely on visibility bitmaps for logical deletion and version management, preventing in-place modifications to data files. This makes it highly susceptible to concurrency coordination conflicts between backend compaction maintenance tasks and frontend delete, update, and write transactions, necessitating a reliable concurrency coordination mechanism to support stable system operation.

[0003] Current mainstream columnar storage engines employ an architecture that combines immutable data files with multi-version concurrency control metadata and visibility bitmaps to achieve data maintenance and transaction commits. After a compaction task completes, the old and new files are replaced by switching file version identifiers. Write transactions, on the other hand, perform logical deletion by writing a deletion flag to the visibility bitmap associated with the source file. To avoid concurrent modifications of the same file by compaction and write transactions, existing technologies often employ coarse-grained locking of the entire table or file group, conflict-triggered transaction rollback and retry, and serial execution of compaction within a limited maintenance window. This implementation logic relies on strong serialization, passive transaction avoidance, or retrying after a conflict as a fallback. It lacks fine-grained processing logic to precisely migrate the writing intent from the old file to the new file, and therefore cannot adapt to the operational requirements of columnar storage engines under high-concurrency mixed loads. Summary of the Invention

[0004] The purpose of this invention is to solve the technical problems of severe blocking during concurrent operation of existing columnar storage engines' Compact tasks and foreground write transactions, easy loss of deletion semantics during version switching, and excessive resource consumption for overall transaction retry.

[0005] The primary technical problem this invention aims to solve is that, during the execution of the Compact task in the background of the columnar storage engine, when concurrent delete, update, and write transactions modify the data rows of the source file, the traditional coarse-grained locking mechanism will block the execution of the foreground business transactions for a long time, resulting in limited system concurrency throughput and causing defects such as write transactions queuing and amplified system tail latency.

[0006] Another technical problem that this invention aims to solve is that during the existing version switching process, the old file data rows are located during the write transaction execution phase, and the source file has been replaced by the new file by the compaction phase. The deletion mark is still written to the invalid old file, causing the new file reading transaction to be unable to perceive the deletion semantics, resulting in the deletion failure and data visibility disorder. Moreover, the overall retry of the transaction triggered by the conflict will repeatedly consume system computing resources, further aggravating the system load pressure.

[0007] The technical solution adopted in this invention is: a method for conflict-free concurrent processing of columnar storage engine compact and write transactions, comprising the following steps: S1. The Compact transaction filters the source files to be merged based on the transaction snapshot and reads the valid data rows inside the source files. S2. Reorganize the valid data rows obtained in S1 to generate a target data file and construct a position mapping relationship from the source row to the target row, and persist the target data file and the position mapping relationship. S3. During the execution phase, the foreground write transaction locates the source file and source line position corresponding to the operation based on its own snapshot, caches the operation intent locally, and does not modify the global visibility state during the execution phase. S4. The Compact transaction enters the commit phase, obtains commit control permissions, detects the deletion information associated with the source file, migrates the committed deletion information to the target data file based on the location mapping relationship, atomically marks the source file as invalid, and publishes a visible version of the target data file. S5. The write transaction enters the commit phase, verifies the effective status of the corresponding source file, and selects to directly write to the source file visibility bitmap or write to the target data file visibility bitmap after converting the row position through the position mapping relationship, and completes the transaction commit. S6. The background asynchronously determines the conditions for recycling mapping relationships based on the global transaction level, and recycles resources for the location mapping relationships that meet the conditions.

[0008] Preferably, step S1 includes: matching the source files visible in the snapshot based on the xmin version identifier and xmax version identifier in the multi-version concurrency control metadata; and comparing and filtering the transaction snapshot version with the file-level version identifier to determine the set of source files to be merged.

[0009] Preferably, step S2 includes: independently constructing the position mapping relationship for each individual source file, recording the correspondence between the original line numbers inside the source file and the line numbers after mapping to the target file; and marking invalid lines that are not included in the target data file as invalid in the position mapping relationship. The target data file and the location mapping relationship are synchronously written to object storage using an atomic persistence protocol, and the target data file remains invisible to the outside world until persistence is completed and version switching occurs.

[0010] Preferably, step S3 includes: the write transaction caches the deletion operation intent only in the local context and the old version logic deletion intent in the update transaction; Step S5 includes: the update transaction only performs row position mapping migration on the old version records, and the new version data is stored by using the incremental write path.

[0011] Preferably, step S4 includes: after the Compact transaction has persisted the target data file and the location mapping relationship, it obtains the commit control permission for the current data shard, and completes the migration of deletion information, the invalidation marking of the source file, and the release of the visible version of the target data file within a controlled time period.

[0012] Preferably, steps S4 and S5 form a two-way adaptive interaction for transaction commit: during the commit phase, the Compact transaction only migrates the committed global visibility state of the source file to the target data file; during the commit phase, the write transaction autonomously selects the write path based on the real-time status of the source file. If it determines that the source file has expired, it converts its own private cache operation intent by reading the position mapping relationship and writes it to the visibility bitmap of the target data file.

[0013] Preferably, step S6 includes: determining the timing of location mapping relationship recycling based on the global minimum active transaction identifier as the water level, and asynchronously recycling the location mapping relationship and its metadata records when there are no active transactions that depend on the corresponding old source file and mapping relationship.

[0014] This invention also discloses a conflict-free concurrent processing system for columnar storage engine Compact and write transactions, comprising: The Compact processing module is used to filter source files to be merged based on transaction snapshots and read valid data rows, reorganize the valid data rows to generate target data files and build position mapping relationships, obtain commit control permissions during the commit phase, migrate deletion information to the target data file based on the position mapping relationship, atomically mark the source files as invalid and publish a visible version of the target data file; The write transaction processing module is used to locate the source file and source line position corresponding to the operation based on its own snapshot during the execution phase and cache the operation intent locally. During the execution phase, it does not modify the global visibility state. During the commit phase, it verifies the effective status of the corresponding source file and chooses to write directly to the source file visibility bitmap or write to the target data file visibility bitmap after converting the line position through the position mapping relationship, depending on the source file status. The mapping and reclamation module is used to asynchronously determine the reclamation conditions based on the global transaction level and perform resource reclamation.

[0015] Preferably, the Compact processing module and the write transaction processing module form a bidirectional adaptive interaction for transaction commit: During the commit phase, the Compact processing module only migrates the committed global visibility state of the source file to the visibility bitmap of the target data file. During the commit phase, the write transaction processing module autonomously selects the write path based on the real-time status of the source file. If it determines that the source file has expired, it reads the position mapping relationship, transforms its own private cache operation intent, and writes it to the visibility bitmap of the target data file.

[0016] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for conflict-free concurrent processing of columnar storage engine Compact and write transactions.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: This invention can eliminate the reliance on coarse-grained locks in the entire Compact process, enabling the background merging task and the foreground write transaction to run concurrently during the execution phase. Only short-term resource coordination is performed at critical nodes of transaction commit, which greatly reduces the scope of system lock contention and blocking time, and improves the overall concurrent processing capability of the columnar storage engine in mixed load scenarios.

[0018] This invention establishes a new and old file line position conversion channel by constructing a line-level mapping relationship. After the source file completes the version failure switch, the modification intention of the write transaction can be accurately migrated to the effective target file, maintaining data consistency and transaction isolation under multi-version concurrency control semantics, and avoiding the phenomena of deletion failure and visibility logic disorder within the version switch window.

[0019] This invention does not require disruptive modifications to the underlying storage architecture and transaction kernel of the columnar storage engine. It is compatible with existing engineering implementation logic such as immutable file storage, visibility bitmap logical deletion, and incremental version writing. At the same time, it relies on global transaction watermarks to achieve automated asynchronous recycling of mapped files, and has good engineering scalability and scenario adaptability, adapting to application requirements such as partition expansion, file size growth, and continuous operation of high-concurrency transactions. Attached Figure Description

[0020] Figure 1 This is the overall flowchart of the non-conflicting concurrent processing method for the columnar storage engine Compact and write transactions of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings and specific examples. It should be noted that, unless otherwise specified, the embodiments and technical features in these embodiments can be combined with each other.

[0022] Example 1 A method for conflict-free concurrent processing of columnar storage engine compact and write transactions includes the following steps: S1. The Compact transaction filters the source files to be merged based on the transaction snapshot and reads the valid data rows inside the source files. S2. Reorganize the valid data rows obtained in S1 to generate a target data file and construct a position mapping relationship from the source row to the target row, and persist the target data file and the position mapping relationship. S3. During the execution phase, the foreground write transaction locates the source file and source line position corresponding to the operation based on its own snapshot, caches the operation intent locally, and does not modify the global visibility state during the execution phase. S4. The Compact transaction enters the commit phase, obtains commit control permissions, detects the deletion information associated with the source file, migrates the deletion information to the target data file based on the location mapping relationship, atomically marks the source file as invalid, and publishes a visible version of the target data file. S5. The write transaction enters the commit phase, verifies the effective status of the corresponding source file, and selects to directly write to the source file visibility bitmap or write to the target data file visibility bitmap after converting the row position through the position mapping relationship, and completes the transaction commit. S6. The background asynchronously determines the conditions for recycling mapping relationships based on the global transaction level, and recycles resources for the location mapping relationships that meet the conditions.

[0023] Furthermore, step S1 also includes S1-1 based on the multi-version concurrency control metadata. , The version identifier matches the source file where the snapshot is visible, specifically: based on the transaction snapshot version. By comparing and filtering with file-level version identifiers, a visible source file discrimination model is constructed: in, For the first A function to determine the visibility of a source file; For logical indicator functions; For the first Create a transaction version from each source file; For the first One source file invalid transaction version; This is the current Compact transaction snapshot version.

[0024] Furthermore, step S2 also includes S2-1, independently constructing row-level mapping files of integer array structure for each individual source file, specifically: establishing a non-linear mapping relationship model from source line numbers to target line numbers: in, These are the original line numbers within the source file; The line numbers are mapped to the target file; A unique identifier for the source file; Used as a unique identifier for the target file; Transformation operator for mapping line positions across files; This is the row validity flag coefficient, with a value of 1 for valid rows and a value of 0 for invalid rows.

[0025] Furthermore, step S3 also includes S3-1 where the write transaction only records the deletion and update of the old version logic deletion intent in the local context, without writing data to the global visibility bitmap.

[0026] The quantitative allocation process of local buffer load is as follows: The system counts the total number of data rows to be operated on in the current write transaction, obtains the corresponding file identifier and source line position for each row to be operated on, and calculates the overhead of the file dimension and the row position dimension according to the preset weight normalization coefficient; by accumulating the overhead of all rows, the local buffer transaction operation load is obtained, and the system dynamically adjusts the memory allocation space of the local transaction buffer accordingly.

[0027] Specifically, this includes constructing a transaction private buffer load representation model: in, Load of local buffer transaction operations; The total number of rows of data to be operated on in a single transaction; For the first The line corresponds to the file identifier; For the first Source line position; This is the weighted normalization coefficient for file identifier and line position.

[0028] Furthermore, step S4 also includes S4-1 utilizing short-term controllable resources to implement short-term serial scheduling during the submission phase.

[0029] The dynamic constraint process for submitting resource occupancy duration is as follows: The metadata management module obtains the preprocessing time of a single deletion information migration and the number of source files to be switched, and calculates the basic processing time in combination with the current system's partition parallel scheduling granularity; at the same time, it dynamically adjusts the environmental adaptive correction coefficient according to the current system's CPU load and object storage IO load, and multiplies the basic processing time by the correction coefficient to obtain the final short-term resource occupancy duration threshold; the metadata management module strictly controls the resource holding period according to this threshold, and automatically releases the resource once it exceeds the threshold.

[0030] Specifically, this includes building a resource occupancy duration constraint model: in, For short-term resource usage duration; Preprocessing time for deleting information migration; The number of source files to be switched; For partitioned parallel scheduling granularity; This is the environmental adaptive correction coefficient.

[0031] Furthermore, steps S4 and S5 form a two-way adaptive interaction for transaction commit. When the Compact transaction completes the version switch first, the existing write transaction operation intention is synchronized to the target data file visibility bitmap in advance. When the write transaction enters the commit stage first, the persisted row-level mapping file is read to complete the row position conversion and writing.

[0032] Furthermore, step S5 also includes S5-1 where the update transaction only performs row position mapping migration on the old version records, and the new version data is stored using the incremental write path.

[0033] The process of splitting and routing update transaction behavior is as follows: When processing update transactions, the system breaks down its behavior into two independent paths; for logical deletion of old version records, the row position mapping transformation operator is reused to complete the position conversion from the source row to the target row and write the deletion intention; for writing new version data, the incremental file direct write operator is called to write the new data directly to the system's preset incremental writable file, thereby skipping the position mapping transformation process.

[0034] Specifically, this includes constructing a formula for determining the breakdown of update transaction behavior: in, To fully update the transaction behavior representation; Remove the behavior component from the old version logic; Write behavioral components to the new version of the data; For reused row position mapping transformation operators; This is an incremental file write-through operator.

[0035] Furthermore, step S6 also includes S6-2 determining the timing of row-level mapping file recycling based on the global minimum active transaction identifier.

[0036] The safety determination process for mapping file reclamation is as follows: The system obtains the global minimum active transaction version in real time as the water level. If the Compact transaction version that generated the mapping relationship is still less than the global minimum active transaction version after adding the product of the system version step interval and the safety water level buffer coefficient, it means that no active transaction currently depends on the old source file and mapping relationship, and reclamation is allowed (output flag is 1); otherwise, reclamation is not performed for the time being (output flag is 0).

[0037] Specifically, this includes constructing a threshold model for determining the recycling of mapped files: in, A permission flag for reclaiming mapped files; Compact transaction version for generating mapping files; This is the minimum active transaction version globally. This is the system version step interval; This is the safety water level buffer coefficient.

[0038] This embodiment, based on the columnar storage engine's immutable file storage architecture and multi-version concurrency control mechanism, fully implements the columnar storage engine's conflict-free concurrent processing method for Compact and write transactions. The overall operational architecture relies on the collaborative work of a metadata management module, a Compact background scheduling module, a foreground transaction processing module, a mapped file management module, a visibility bitmap maintenance module, and a background resource reclamation module. Figure 1 This diagram illustrates the overall architecture for conflict-free concurrent processing of the columnar storage engine Compact and write transactions. Within the architecture, each module interacts with the others through a metadata bus, data read / write channels, and version synchronization channels to exchange data and transmit instructions.

[0039] During step S1, the Compact transaction filters the source files to be merged based on the transaction snapshot, reads the valid data rows inside the source files, and simultaneously matches them with the metadata in S1-1 based on multi-version concurrency control. , The version identifier is matched with the source file that is visible in the snapshot, and the file filtering is completed based on the visible source file discrimination model.

[0040] The Compact backend scheduling module first creates an independent transaction snapshot to lock the current system static version view, thus preventing the subsequent file filtering process from being disturbed by data from concurrent write transactions in the foreground.

[0041] Multi-version concurrency control metadata is uniformly stored within the global metadata management module, and each source file is bound to an independent metadata management module. Version identifier and Version identifiers and global metadata management modules are arranged in an orderly manner according to file partition and file generation time dimensions.

[0042] The specific judgment process is as follows: Get the current Compact transaction snapshot version; and read the xmin and xmax version identifiers of the source file; if the snapshot version is greater than or equal to xmin and less than xmax, the source file is considered visible and included in the set to be merged; otherwise, it is not included.

[0043] Specifically, this includes the following visible source file discrimination model. The computation is loaded and executed by the Compact background scheduling module, and the logical indicator function is used. Output the logical judgment result. The judgment result is limited to 0 and 1. When the value is 1, it means that the corresponding source file is included in the set to be merged. When the value is 0, it means that the corresponding source file will not participate in the Compact merging process for the time being.

[0044] The Compact background scheduling module traverses all data files under the partition, and substitutes them one by one into the visible source file discrimination model to complete batch filtering. After the filtering is completed, the first data file set is formed, which is the set of source files to be merged.

[0045] The Compact background scheduling module initiates a disk data read command, reads the original data pages of all source files within the first data file set from the object storage carrier, parses each data row according to the columnar data organization format, filters out valid data rows that match the visible range of the current transaction snapshot, and caches all valid data rows in the Compact private memory buffer to provide basic data input for the subsequent step S2.

[0046] During step S2, the valid data rows obtained in S1 are sorted, encoded, and compressed to generate a target data file and construct a row-level mapping file from the source row to the target row. The target data file and the row-level mapping file are persisted. At the same time, the row-level mapping file of an integer array structure is independently constructed according to each source file in S2-1, and the row position association is established based on the nonlinear mapping relationship model.

[0047] The Compact backend scheduling module obtains the valid data rows cached in the private memory buffer in step S1, and completes the global sorting process according to the business preset sorting dimensions, including but not limited to partition key, sort key, and timestamp field.

[0048] After sorting, columnar recoding and data compression encoding are performed sequentially. The encoding and compression algorithm uses the built-in compression operator of the columnar storage engine. After processing, a brand new target data file is generated according to the columnar storage file organization structure.

[0049] The mapping file management module creates a row-level mapping file with a corresponding integer array structure for each source file within the first data file set. The storage format of the row-level mapping file is bound to the source file one by one.

[0050] The nonlinear mapping relationship model The operation is performed by the mapping file management module, which is a cross-file line position mapping transformation operator. Unique identifier of merged source file Unique identifier of target file Original line numbers within the source file Three parameters are used to output the line numbers after mapping the initial target file. .

[0051] Row validity labeling coefficient The initial mapping result is corrected. The coefficient corresponding to the valid data line in the source file is set to 1, and the coefficient corresponding to the invalid line not included in the target data file is set to 0. When the coefficient is 0, the corresponding mapping position is marked as invalid.

[0052] The mapping file management module writes the correspondence between each set of source rows and target rows into the row-level mapping file. After all mapping relationships are entered, the target data file and all row-level mapping files are synchronously written to object storage through the atomic persistence protocol. After persistence is completed, the visibility state of global metadata is not modified, and the target data file remains invisible to the outside world.

[0053] Furthermore, to address the issue of excessive space consumption caused by the location mapping relationship when the source file has tens of millions of lines, one feasible implementation supports compressed storage or segmented management of the mapping relationship; for example, run-length encoding can be used to compress and record consecutive invalid line intervals; or the mapping relationship can be split into multiple segmented mapping files at the line group granularity, and the corresponding segments can be loaded as needed when the write transaction is committed, so as to reduce the memory consumption overhead per transaction.

[0054] Specifically, the nonlinear mapping relationship model includes the following: for each line in the source file, if the line is a valid data line with a coefficient of 1, its new line number in the target file is calculated and recorded using the mapping operator; if the line is an invalid line with a coefficient of 0, it is marked as invalid in the mapping relationship, such as by setting it to -1 or a specific identifier.

[0055] In this embodiment, the position mapping relationship is specifically implemented using a row-level mapping file with an integer array structure; In one feasible embodiment, the location mapping relationship also includes using data structures such as hash tables, B-tree indexes, or key-value pairs to record the correspondence between the source row and the target row, all of which fall within the protection scope of this invention.

[0056] During the execution of step S3, the foreground write transaction locates the source file and source line position corresponding to the operation based on its own snapshot during the execution phase, caches the operation intent in the local transaction buffer, and matches the S3-1 write transaction to record the deletion and update of the old version logic deletion intent only in the local context, without writing data to the global visibility bitmap, and quantifies the storage overhead by relying on the transaction private buffer load representation model.

[0057] The front-end transaction processing module receives delete and update transaction instructions from the business layer, allocates an independent transaction snapshot for each write transaction, and maintains the same transaction snapshot isolation level as the system's multi-version concurrency control mechanism.

[0058] The front-end transaction processing module traverses the global metadata management module based on the transaction snapshot to locate the source file identifier and source line position of the data row to be operated on, without triggering the immediate write operation of the global visibility bitmap.

[0059] In other words, write transactions only cache the operation intent locally during the execution phase and do not modify the global visibility state during the execution phase.

[0060] The local cache is not limited to the local transaction buffer; it can also be a private transaction rollback log or other storage format that can temporarily store the intention of the operation and is not visible to the outside world.

[0061] Each write transaction is allocated an independent local transaction buffer, which belongs to the transaction's private context space and does not compete with other transactions for memory resources.

[0062] The transaction private buffer load representation model The total number of data rows to be processed in a single transaction is calculated in real time by the front-end transaction processing module. The weighted normalization coefficients of file identifier and row position are used to calculate the number of pending data entries in the current transaction. The configuration is initialized during system startup, and the two coefficients maintain a fixed normalization constraint relationship.

[0063] The front-end transaction processing module calculates the local buffer transaction operation load by accumulating data line by line. The local buffer memory allocation space is dynamically adjusted according to the load, and the file identifier is also set. With source line position The operation intent data structure is formed and fully cached within the local transaction buffer.

[0064] During this phase, the update transaction completes the behavior decomposition, breaking down the complete update behavior into old version logical deletion behavior and new version data writing behavior. Only the old version logical deletion related operation intentions are cached in the local transaction buffer, and the new version data writing behavior reserves the subsequent incremental write channel.

[0065] During step S4, the Compact transaction enters the commit phase, requests short-time commit control resources, detects the deletion information associated with the source file, migrates the deletion information to the target data file based on the row-level mapping file, atomically marks the source file as invalid and publishes a visible version of the target data file, and simultaneously matches S4-1 to realize short-time serial scheduling of the commit phase using short-time controllable resources, and manages the resource occupation cycle based on the commit resource occupation duration constraint model.

[0066] In this process, after the target data file and row-level mapping file are persisted, the Compact background scheduling module enters the transaction commit phase. It requests partition-level short-term commit control resources from the metadata management module. The resource control scope is limited to the partition corresponding to the current merge task, without blocking business read / write operations and background merge tasks in other partitions. The commit resource occupancy duration constraint model is described above. The metadata management module calculates and solves the problem; the time required for preprocessing the deletion information migration is [not specified]. Based on historical task operation statistics, the number of source files to be switched is dynamically collected. Total number of files taken from the first data file set, partition parallel scheduling granularity Preset partition concurrency control parameters for the system, and adjust the environmental adaptive coefficient. The system dynamically and adaptively adjusts based on CPU load and object storage I / O load. The model solves for the short-term resource usage duration. The metadata management module strictly controls the resource holding period according to the duration threshold. If the threshold is exceeded, the resource will be automatically released to avoid long-term blocking.

[0067] The Compact background scheduling module traverses the visibility bitmap associated with the source file, reads the row-level mapping file that has been persisted in step S2, reuses the nonlinear mapping relationship model to complete the conversion of the source row position to the target row position, and synchronously migrates the logical deletion markers that have been committed in the source file to the visibility bitmap corresponding to the target data file.

[0068] After the migration process is complete, the metadata management module performs an atomic metadata update operation, updating all source files. The version identifier is updated to the current Compact transaction version, the source file is marked as entering an invalid state, the metadata state of the target data file is modified, and the target data file is published as a globally visible version. The version switching process uses atomic transactions to ensure no intermediate inconsistencies.

[0069] Furthermore, if a Compact transaction times out or crashes within the controlled duration, the metadata management module will trigger the timeout release mechanism to automatically release the commit control privileges.

[0070] Meanwhile, since the target data file and location mapping relationship remain invisible to the outside world after persistence and before version switching, the system only needs to discard the persisted target data file and mapping relationship, and can complete the transaction rollback without performing complex reverse data migration operations, thus ensuring the strong consistency of the system.

[0071] During the interactive operation of steps S4 and S5, the Compact transaction only migrates the committed global visibility state of the source file to the visibility bitmap of the target data file during the commit phase; the write transaction autonomously selects the write path based on the real-time status of the source file during the commit phase, completely avoiding the isolation disruption caused by premature globalization of uncommitted data.

[0072] If the Compact transaction completes the version switching process before the write transaction, the Compact background scheduling module only reads the persistently committed deletion markers in the source file visibility bitmap, and synchronizes them to the target data file visibility bitmap after the row position conversion is completed by the mapping file management module. This ensures that the target data file inherits the historical valid deletion semantics as soon as it becomes visible to the outside world, but does not collect or interfere with the uncommitted write transaction operation intentions inside the foreground transaction buffer.

[0073] When a write transaction enters the commit phase, the front-end transaction processing module reads the file status flag from the global metadata management module in real time: if it determines that the source file has not undergone a version switch, that is, Compact has not yet completed the commit or has not been executed, it directly writes the locally cached operation intent into the source file visibility bitmap; if it determines that the source file has been marked as invalid, that is, Compact has completed the version switch, it actively pulls the corresponding row-level mapping file, completes the row position conversion based on the mapping relationship, and smoothly migrates its own operation intent into the target data file visibility bitmap.

[0074] Through the above mechanism, the modification intention of a write transaction is strictly kept private before it is committed. Regardless of whether Compact has completed the switch, the write transaction can be accurately implemented according to the current state of the file at the time of commit, avoiding the complexity of repeated application and rollback. The two types of transactions run in full concurrency during the execution phase, and only complete the collaborative adaptation through metadata and mapping files at the key commit nodes.

[0075] During step S5, the write transaction enters the commit phase, verifies the effective status of the corresponding source file, and selects to directly write to the source file visibility bitmap or write to the target data file visibility bitmap after converting the row position through the row-level mapping file, thus completing the transaction commit. At the same time, the update transaction matched by S5-1 only performs row position mapping migration on the old version records, and the new version data is stored using the incremental write path. The behavior decoupling representation is achieved by relying on the update transaction behavior splitting and discrimination formula.

[0076] In this process, after the front-end transaction processing module completes the caching of operation intentions in the local transaction buffer, it enters the transaction commit phase, verifying the source file metadata status corresponding to each operation intention one by one. The metadata status verification is based on the source file... The relationship between the version identifier and the current transaction snapshot version is implemented. When the source file is not expired, the front-end transaction processing module directly writes the locally cached deletion intent into the visibility bitmap associated with the source file. When the source file is expired, the front-end transaction processing module calls the mapping file management module to load the corresponding row-level mapping file, reuses the row position mapping transformation operator to complete the conversion from source row number to target row number, and writes the deletion intent into the visibility bitmap associated with the target data file. The update transaction behavior splitting and discrimination formula... The transaction behavior representation is fully updated by the front-end transaction processing module. Disassemble into old version logic deletion behavior components Data write behavior component of the new version The old version's logic for deleting components reuses existing ones. The row position mapping transformation operator completes the migration process, and the new version of data is written to the row components via... The incremental file direct write operator writes to the system's preset incrementally writable file and does not participate in the row-level mapping conversion process. After all visibility bitmap write operations are completed, the front-end transaction processing module executes the transaction commit log to disk, releases the local transaction buffer memory resources, and marks the write transaction as successfully completed.

[0077] During the execution of step S6, the background asynchronously determines the conditions for reclamation of mapping files based on the global transaction water level, and reclaims resources for row-level mapping files that meet the conditions. At the same time, it matches the timing of row-level mapping file reclamation based on the global minimum active transaction identifier as the water level in S6-2, and completes the reclamation logic discrimination based on the mapping file reclamation judgment threshold model.

[0078] The mapping file management module initiates background asynchronous periodic tasks. These periodic tasks run independently of compact transactions and foreground write transactions, without consuming core business processing resources. The global metadata management module provides real-time statistics on the global minimum active transaction version. System version step interval The transaction version increments by a fixed step size, with a safety water level buffer coefficient. Redundancy protection parameters configured for the system. The mapping file recycling determination threshold model. The mapping file management module performs line-by-line mapping file substitution calculations to generate a mapping file recycling permission flag. A value of 1 indicates that the current row-level mapping file meets the reclamation conditions, while a value of 0 indicates that snapshot transactions still depend on the old source file and mapping relationship, and reclamation operations are not performed temporarily. For row-level mapping files that meet the reclamation conditions, the mapping file management module performs object storage file deletion and metadata record cleanup, synchronously releasing the memory metadata space occupied. The entire reclamation process is executed asynchronously and without locks, without interfering with the normal concurrent operation of foreground write transactions and background compact transactions.

[0079] Furthermore, to address the issue of multi-level mapping nesting caused by the target data file being used as the source file to initiate a new round of compaction while the position mapping relationship has not been reclaimed, this invention actively loads the historical mapping relationship from the previous round when constructing a new round of position mapping relationship, flattening and merging the multi-level mapping into a direct mapping relationship from the latest source file to the final target file, thus avoiding the performance loss of multi-level chain lookup when committing write transactions.

[0080] This invention also provides a columnar storage engine compact and write transaction conflict-free concurrent processing system, which includes: The Compact processing module encompasses the functions of the aforementioned Compact background scheduling module and mapping file management module, and is used to execute the processes of steps S1, S2, and S4. Write the transaction processing module, which is the aforementioned front-end transaction processing module, and execute the processes of steps S3 and S5. The mapping recycling module, namely the background resource recycling module mentioned above, is used to execute the process of step S6.

[0081] The Compact processing module and the write transaction processing module form a bidirectional adaptive interaction for transaction commit: during the commit phase, the Compact processing module only migrates the committed global visibility state of the source file to the visibility bitmap of the target data file; during the commit phase, the write transaction processing module autonomously selects the write path based on the real-time status of the source file. If it determines that the source file has expired, it converts its own private cache operation intent by reading the position mapping relationship and writes it to the visibility bitmap of the target data file.

[0082] Example 2 This embodiment is applied to a mixed workload scenario of cloud-native columnar storage database under a storage-compute separation architecture. The scenario simultaneously carries real-time online update and delete business and offline massive data query and analysis business. The background periodically triggers the Compact merging and compression task, while the foreground continuously receives Delete and Update transactions with high concurrency. Traditional coarse-grained locking methods will cause problems such as business write blocking, increased transaction retry overhead, and loss of deletion semantics during version switching. The columnar storage engine Compact and write transaction conflict-free concurrent processing method described in this invention can adapt to this complex workload scenario.

[0083] In a compute-storage separation architecture, data files are uniformly stored in remote object storage. Compute nodes have no local persistent storage capabilities. All source file reading, target file generation, and row-level mapping file persistence are completed through the object storage standard read / write interface. Internally, each compute node deploys four core processes: metadata management, compact scheduling, transaction management, and mapping file management. These processes communicate with each other via shared memory channels, transmitting core data such as transaction snapshot versions, file version identifiers, row position mapping relationships, and buffer load parameters. Within each process, calculations and judgments are strictly performed according to the aforementioned formula models.

[0084] The Compact scheduling process executes step S1 according to a fixed cycle, using the visible source file discrimination model to batch filter columnar data files to be merged in object storage. The filtering process relies entirely on... , The interval constraint relationship with the transaction snapshot version is used to filter out the source files to be merged, which include small-granularity row group files generated by long-term business writes and expired incremental files. After the compute node pulls the file data pages from the object storage, it parses and extracts the snapshot to see the valid data rows.

[0085] Step S2 completes the sorting, recoding, and compression of valid data rows in the memory of the computing node. The generated target data file is written to object storage according to the columnar storage standard format. At the same time, the mapping file management process generates a row-level mapping file with an integer array structure for each remote source file. The binding of source rows and target rows is completed through the nonlinear mapping relationship model. Invalid rows are uniformly marked with special coefficient status. The mapping file and the target data file are written to object storage using an atomic batch persistence mechanism to ensure the consistency of version timing between the two.

[0086] The Delete and Update transactions issued by the massive online business on the front end are handled by the transaction management process. During the transaction execution phase, the S3 logic is executed. An independent isolated snapshot is allocated to each write transaction. After locating the source file and source line position to be operated on, the operation intention is cached only in the process's private local transaction buffer. The buffer memory capacity is dynamically adjusted through the load representation model of the transaction private buffer. No write requests are initiated to the global visibility bitmap, so as to achieve completely lock-free concurrency between the execution phase and the background Compact task.

[0087] When the Compact task enters the commit phase and executes step S4, the scheduling process requests partition-level short-term commit control resources. It precisely controls the resource holding time using the commit resource occupancy duration constraint model to avoid prolonged occupancy causing business blockage. The scheduling process reads the persisted row-level mapping file, migrates the committed historical deletion information associated with the source file to the target data file visibility bitmap, without interfering with uncommitted write transactions. Then, it uses a metadata atomic update mechanism to mark the old source file as invalid and the new target file as globally visible. The entire version switching process is free of data inconsistency windows.

[0088] At the transaction commit interaction level, the bidirectional adaptation logic of steps S4 and S5 is strictly followed. Under high-concurrency mixed loads, there are many scenarios where compact transactions and write transactions are committed alternately. If compact transactions are committed first, the deletion semantics are pre-synchronized to the new file. If write transactions are committed first, the mapping file is actively loaded to complete the row position conversion, without the need for transaction rollback and retry, or locking and blocking the entire table. When the write transaction executes the commit logic of step S5, the write path is selected by judging the branch through metadata status. The update transaction relies on the above update transaction behavior splitting judgment formula to separate the old version migration logic and the new version incremental write logic, ensuring that the update semantics are complete and not lost.

[0089] The background mapping file management process persistently executes step S6 asynchronous recycling logic, which collects the global minimum active transaction version in real time, substitutes it into the mapping file recycling judgment threshold model to determine the recycling eligibility of each row-level mapping file, and asynchronously cleans up the mapping files and corresponding metadata records in the object storage when there are no historical snapshot transactions that depend on the old file mapping relationship, continuously releasing storage and metadata management resources, and adapting to the operation and maintenance needs of massive files running for a long time in cloud-native scenarios.

[0090] In this application scenario, the entire method abandons traditional table-level locks, file set coarse-grained locks, and off-peak serialization avoidance mechanisms. It relies on row-level mapping files to build an intermediate bridge for version switching, converging conflict constraints to the short-term phase of transaction commit. The execution phase runs with high concurrency throughout. At the same time, it quantifies six core logics—file visibility, row position mapping, buffer load, lock occupancy duration, update behavior splitting, and mapping file reclamation—through multiple sets of formula models. It adapts to the engineering operation requirements of storage-compute separation, object storage, and mixed high-concurrency loads. It is compatible with the native architecture of column-oriented storage engines with immutable files, logical deletion, and MVCC version management. It does not require disruptive modifications to the database kernel storage layer and transaction layer, and has strong scenario adaptability and engineering feasibility.

[0091] Example 3 This embodiment is applied to a partitioned parallel compaction operation and maintenance scenario of a big data analysis columnar database. The database is divided into multiple partitions according to time and business dimensions. Each partition independently performs data writing, version management, and background compaction merging tasks. The partitions are isolated from each other but share a global transaction version level and metadata management system. It is very easy for large-scale conflicts to occur between cross-partition compaction tasks and high-concurrency write transactions within a partition. The technical method of this invention can achieve conflict-free concurrency within a partition and independent scheduling and operation between partitions.

[0092] The partition compact scheduling submodule independently executes step S1. Based on the partition's local transaction snapshot, it calls the visible source file discrimination model to filter the source files to be merged within the partition. All model calculation parameters are taken from the partition's local file metadata and the partition's transaction snapshot version. The file filtering logic between partitions is independent, preventing cross-partition file version interference. The resulting set of source files to be merged only includes columnar data files under the current partition. The effective data row reading and parsing process is limited to the partition's private memory space, avoiding memory resource contention among multiple partitions.

[0093] When the partition compact scheduling submodule executes step S2, it completes the sorting, encoding, and compression of valid data rows in the partition's private memory to generate a partition-specific target data file. The partition mapping file management submodule constructs an independent row-level mapping file for each source file within the partition. It establishes a mapping association between source rows and target rows within the partition through the nonlinear mapping relationship model. The mapping file naming format carries a unique partition identifier, enabling partition-isolated management of mapping files globally. The persistent storage path is divided according to the partition directory hierarchy, facilitating subsequent loading and reclamation.

[0094] Real-time business write transactions within a partition are handled by the partition transaction processing submodule. During step S3, the source files and line positions within the partition are located based on the partition isolation snapshot. The operation intent is cached in the partition's private transaction buffer. The total load of the transaction buffer within the partition is quantified using the load characterization model of the private transaction buffer. The global metadata management center summarizes the load data of each partition in real time, achieving dynamic balancing of global memory resources. Update transactions within a partition are uniformly decomposed into logical deletion of the old version and incremental writing of the new version. Only the operation intent of the old version is included in the buffer cache, and the new version data is written to the partition's dedicated incremental file directory.

[0095] When the partition compact task enters step S4 (commit phase), it only requests partition-level short-term commit control resources and does not occupy global-level lock resources. The partition resource holding duration is calculated using the commit resource holding duration constraint model. The partition parallel scheduling granularity in the model adapts to the current number of concurrent tasks in the partition, and the environment adaptive correction coefficient is independently adjusted according to the partition load, achieving differentiated management of inter-partition lock control duration. The partition compact scheduling submodule relies on the partition row-level mapping file to migrate the source file deletion information within the partition to the visibility bitmap of the partition's target data file, atomically updating the partition file metadata. , Version identification enables seamless switching between file versions within a partition.

[0096] Each partition strictly follows the bidirectional commit interaction rules of steps S4 and S5. Compact transactions and write transactions within a partition adapt autonomously according to their own commit sequence, without the need for global unified scheduling. When the partition transaction processing submodule executes the commit logic of step S5, it only verifies the metadata status of the source file within the partition. Based on the status, it chooses to directly write to the visibility bitmap of the partition source file or write to the visibility bitmap of the target file after conversion through the partition row-level mapping file. Update transactions complete behavior decoupling through the update transaction behavior splitting and discrimination formula. The old version mapping migration and the new version partition incremental direct write do not interfere with each other.

[0097] The global transaction version statistics center calculates the minimum active transaction version globally in real time, a parameter shared by all partitions. The global management center uniformly executes step S6 asynchronous recycling process, substituting the aforementioned mapping file recycling judgment threshold model to determine the recycling conditions for each partition. The safety water level buffer coefficient and system version stepping interval are globally unified configuration parameters, ensuring a consistent standard for multi-partition mapping file recycling logic. Partition mapping files that meet the recycling conditions are asynchronously cleaned up by the global management center. The cleanup operation is performed in batches according to the partition directory, without affecting the normal operation of compact tasks and write transactions in other partitions.

[0098] This embodiment is adapted to columnar database architecture with multi-partition parallel operation and maintenance. Through partition-isolated task scheduling, metadata management, mapping file storage, and transaction buffer control, combined with the standardized calculation logic of the multi-formula model of this invention, it achieves conflict-free concurrency of compact and write transactions within a single partition and independent operation between multiple partitions without interference. It reduces the scope of lock contention and converges conflict boundaries from the partition dimension, while retaining the ability to uniformly manage global version levels. It effectively solves the technical problems of coarse-grained lock contention, conflict range diffusion, and high metadata management complexity in large-scale partition scenarios, and is fully in line with the business operation characteristics of partition operation and maintenance of big data analytical databases.

Claims

1. A method for conflict-free concurrent processing of columnar storage engine compact and write transactions, characterized in that, Includes the following steps: S1. The Compact transaction filters the source files to be merged based on the transaction snapshot and reads the valid data rows inside the source files. S2. Reorganize the valid data rows obtained in S1 to generate a target data file and construct a position mapping relationship from the source row to the target row, and persist the target data file and the position mapping relationship. S3. During the execution phase, the foreground write transaction locates the source file and source line position corresponding to the operation based on its own snapshot, caches the operation intent locally, and does not modify the global visibility state during the execution phase. S4. The Compact transaction enters the commit phase, obtains commit control permissions, migrates the committed deletion information in the source file to the visibility bitmap of the target data file based on the position mapping relationship, atomically marks the source file as invalid and publishes the visible version of the target data file; S5. The write transaction enters the commit phase. It verifies whether the corresponding source file has been marked as invalid. Based on the source file status, it chooses to directly write to the source file visibility bitmap or to write to the target data file visibility bitmap after converting the row position through the position mapping relationship, and completes the transaction commit. S6. The background asynchronously determines the conditions for recycling mapping relationships based on the global transaction level, and recycles resources for the location mapping relationships that meet the conditions.

2. The method for conflict-free concurrent processing of columnar storage engine compact and write transactions according to claim 1, characterized in that, Step S1 includes: matching the source files visible in the snapshot based on the xmin version identifier and xmax version identifier in the multi-version concurrency control metadata; and comparing and filtering the transaction snapshot version with the file-level version identifier to determine the set of source files to be merged.

3. The method for conflict-free concurrent processing of columnar storage engine compact and write transactions according to claim 1, characterized in that, Step S2 includes: independently constructing the position mapping relationship for each individual source file, recording the correspondence between the original line numbers inside the source file and the line numbers after mapping to the target file; and marking invalid lines that are not included in the target data file as invalid in the position mapping relationship. The target data file and the location mapping relationship are synchronously written to object storage using an atomic persistence protocol, and the target data file remains invisible to the outside world until persistence is completed and version switching occurs.

4. The method for conflict-free concurrent processing of columnar storage engine compact and write transactions according to claim 1, characterized in that, Step S3 includes: the write transaction caches the deletion operation intent only in the local context and the old version logic deletion intent in the update transaction; Step S5 includes: the update transaction only performs row position mapping migration on the old version records, and the new version data is stored by using the incremental write path.

5. The method for conflict-free concurrent processing of columnar storage engine compact and write transactions according to claim 1, characterized in that, Step S4 includes: after the Compact transaction has persisted the target data file and the location mapping relationship, it obtains the commit control permission for the current data shard, and completes the migration of deletion information, the invalidation marking of the source file, and the release of the visible version of the target data file within the controlled time.

6. The method for conflict-free concurrent processing of columnar storage engine compact and write transactions according to claim 1, characterized in that, During the commit phase, a Compact transaction only migrates the committed global visibility state of the source file to the visibility bitmap of the target data file. During the commit phase, the write transaction autonomously selects the write path based on the real-time status of the source file. If it determines that the source file has expired, it reads the location mapping relationship, transforms the operation intent of its own private cache, and writes it to the visibility bitmap of the target data file.

7. The method for conflict-free concurrent processing of columnar storage engine compact and write transactions according to claim 1, characterized in that, Step S6 includes: using the global minimum active transaction identifier as the water level to determine the timing of location mapping relationship recycling; when there are no active transactions that depend on the corresponding old source file and mapping relationship, asynchronously recycling the location mapping relationship and its metadata records.

8. A concurrent processing system for columnar storage engine Compact and write transactions without conflict, characterized in that, include: The Compact processing module is used to filter source files to be merged based on transaction snapshots and read valid data rows, reorganize the valid data rows to generate target data files and build position mapping relationships, obtain commit control permissions during the commit phase, migrate deletion information to the visibility bitmap of the target data file based on the position mapping relationship, atomically mark the source files as invalid and publish the visible version of the target data file; The write transaction processing module is used to locate the source file and source line position corresponding to the operation based on its own snapshot during the execution phase and cache the operation intent locally. During the execution phase, it does not modify the global visibility state. During the commit phase, it verifies whether the corresponding source file has been marked as invalid and selects to write directly to the source file visibility bitmap or write to the target data file visibility bitmap after converting the line position through the position mapping relationship, depending on the source file status. The mapping and reclamation module is used to asynchronously determine the reclamation conditions based on the global transaction level and perform resource reclamation.

9. A concurrent processing system for columnar storage engine Compact and write transactions without conflict, as described in claim 8, is characterized in that... The Compact processing module and the write transaction processing module form a bidirectional adaptive interaction for transaction commit: During the commit phase, the Compact processing module only migrates the committed global visibility state of the source file to the target data file visibility bitmap; During the commit phase, the write transaction processing module autonomously selects the write path based on the real-time status of the source file. If it determines that the source file has expired, it converts the operation intent of its own private cache by reading the location mapping relationship and writes it to the visibility bitmap of the target data file.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a conflict-free concurrent processing method for columnar storage engine Compact and write transactions as described in any one of claims 1 to 7.

Images