Database query acceleration method and device based on PCIe interface

Abstract

The application relates to the technical field of database processing and hardware acceleration, and provides a database query acceleration method and device based on a PCIe interface, which comprises the following steps: obtaining a query plan for target data and identifying a hardware-executable query segment from the query plan; determining a data moving cost index and a result compression benefit index corresponding to the query segment and determining a hardware unloading evaluation value according to the data moving cost index and the result compression benefit index; in response to the hardware unloading evaluation value satisfying a preset condition, generating an acceleration descriptor containing transaction version check information and sending the acceleration descriptor to a hardware acceleration device; receiving a compressed intermediate result returned by the hardware acceleration device after the hardware acceleration device performs consistency check on a data block based on the transaction version check information, and generating a query response based on the compressed intermediate result. The application realizes fine-grained and accurate unloading which takes into account transmission efficiency and transaction consistency.

Description

Technical Field

[0001] This application relates to the field of database processing and hardware acceleration technology, specifically to a database query acceleration method and apparatus based on the PCIe interface. Background Technology

[0002] As database data volumes continue to grow, database servers often face challenges such as high CPU resource consumption, limited memory bandwidth, and increased I / O wait times when performing complex operations like large table scans, multi-condition filtering, aggregation statistics, and join queries. While existing technologies utilize hardware devices such as FPGAs or dedicated accelerator cards to accelerate database queries, they still have shortcomings in practical applications. Firstly, current solutions typically consider only a single dimension, such as data size or selectivity, when deciding whether to offload tasks to hardware, neglecting the transmission overhead incurred by moving data between the host and hardware. This can lead to situations where, when the query result set is large or PCIe bandwidth is limited, the data transfer cost may outweigh the benefits of hardware computation, resulting in ineffective or low-return offloading. Secondly, when hardware devices read underlying data blocks via direct memory access, the lack of a consistency verification mechanism with the database transaction view can cause inconsistencies between the data version processed by the hardware and the current transaction snapshot, thus affecting the correctness of the query results. Therefore, there is an urgent need for a database query hardware acceleration solution that can comprehensively consider data transmission costs and computational benefits while ensuring transaction consistency. Summary of the Invention

[0003] To address the issues of existing technologies where data transfer overhead during hardware acceleration can easily offset computational gains and transaction consistency is difficult to guarantee, this application proposes a hardware acceleration method for database queries that achieves fine-grained and precise offloading while balancing transmission efficiency and transaction consistency.

[0004] To achieve the above objectives, this application adopts the following technical solution: Obtain a query plan for the target data, and identify hardware-executable query segments from the query plan; Determine the data migration cost index and result compression benefit index corresponding to the query segment; Based on the data migration cost index and the result compression benefit index, determine the hardware offloading evaluation value; In response to the hardware offload evaluation value meeting the preset conditions, an acceleration descriptor containing transaction version verification information is generated and the acceleration descriptor is sent to the hardware acceleration device. The system receives the intermediate compression result returned by the hardware acceleration device and generates a query response based on the intermediate compression result; wherein the intermediate compression result is generated by the hardware acceleration device after performing consistency verification on the acquired data blocks based on the transaction version verification information.

[0005] As one implementation method, the data migration cost index is determined based on the data reading transmission overhead and the data return transmission overhead; the result compression benefit index is determined based on the estimated original result data volume and the estimated compressed result data volume.

[0006] As one implementation, the host processing load status and hardware pipeline load status are obtained; based on the host processing load status and the hardware pipeline load status, the computation time saving, relocation increase time, and compression time saving are determined; based on the computation time saving, the relocation increase time, and the compression time saving, the hardware offload evaluation value is calculated using the following formula: in, The hardware unloading evaluation value, To save time in the calculation, Add time to the relocation. This saves time for the compression. The time saved in calculation is determined based on the host processing load status and the estimated computational load of the query segment; the time increase in relocation is determined based on the data read transmission overhead, the data return transmission overhead, and the hardware pipeline load status; the time saved in compression is determined based on the estimated raw result data volume, the estimated compressed result data volume, and the bus bandwidth.

[0007] In one implementation, the acceleration descriptor further includes result return format indication information; the compressed intermediate result is intermediate data of an incomplete result set; receiving the compressed intermediate result returned by the hardware acceleration device includes: receiving the intermediate data generated by the hardware acceleration device according to the result return format indication information.

[0008] In one implementation, the intermediate state data includes at least one of the following: when the query segment is a filtering query, the intermediate state data is a filtering bitmap; when the query segment is an aggregation query, the intermediate state data is an aggregation intermediate value; when the query segment is a join query, the intermediate state data is a join matching index; when the query segment is a sorting query, the intermediate state data is a sorting index or a Top-N candidate set.

[0009] As one implementation method, a transaction snapshot identifier is dynamically generated based on the current transaction view; the transaction snapshot identifier is used as the transaction version verification information and encapsulated in the acceleration descriptor; wherein, the transaction snapshot identifier includes at least one of a data page version number or a log sequence number.

[0010] As one implementation, query segments containing at least one of the following operators are identified from the query plan as hardware-executable query segments: data block scan operator, predicate filtering operator, field projection operator, expression calculation operator, hash aggregation operator, semi-join operator, hash join operator, Top-N filtering operator, and local sorting operator; software execution segments other than the hardware-executable query segments in the query plan are retained; the compressed intermediate results are merged with the execution results of the software execution segments to generate the query response.

[0011] Furthermore, this application also provides a hardware acceleration method for database queries, applied to a hardware acceleration device, comprising: receiving an acceleration descriptor sent by a host, the acceleration descriptor including transaction version verification information and result return format indication information; obtaining a data block based on the acceleration descriptor through a direct memory access channel; performing a consistency check on the data block based on the transaction version verification information; in response to the consistency check passing, performing hardware query processing on the data block to generate a compressed intermediate result of an incomplete result set; and returning the compressed intermediate result to the host according to the result return format indication information.

[0012] In one implementation, in response to the failure of the consistency check, a reread instruction or a software execution instruction is returned to the host; hardware query processing is performed on the data block, including: on-chip data rearrangement of the data block to convert it into a data stream adapted to the hardware pipeline; and query processing is performed on the data stream through the hardware pipeline.

[0013] In addition, this application also provides a hardware acceleration system for database queries, including: a database server configured to execute the host-side method as described above; and a hardware acceleration device connected to the database server via a peripheral interconnect bus and configured to execute the hardware-side method as described above.

[0014] Beneficial effects: The technical solution provided in this application determines the hardware offloading evaluation value by jointly using data migration cost indicators and result compression benefit indicators, thus overcoming the limitations of existing technologies that rely solely on data scale or selectivity for offloading decisions. Its principle lies in incorporating the read and return overhead of buses such as PCIe into the benefit calculation model, and comprehensively weighing the bandwidth savings brought by compressing intermediate results. This avoids ineffective offloading in scenarios with large data return volumes or bus congestion, ensuring that the actual net benefit of hardware acceleration is positive.

[0015] This application solves the technical problem that direct hardware reading of underlying data may compromise transaction consistency by carrying transaction version verification information in the acceleration descriptor and having the hardware acceleration device perform consistency verification on the acquired data blocks based on this information. The principle is to establish a version verification mechanism synchronized with the database transaction view on the hardware side using identifiers such as data page version numbers or log sequence numbers. This ensures that the data processed by the hardware strictly conforms to the current transaction snapshot, thereby achieving secure hardware acceleration while guaranteeing ACID properties.

[0016] This application reduces the amount of data transfer between the host and the hardware acceleration device by generating and returning compressed intermediate results of incomplete result sets. The principle is to customize the generation of high-density data structures such as bitmaps, intermediate values, and matching indexes based on the query operator type (e.g., filtering, aggregation, join, sorting), replacing the traditional mode of returning complete result tables. This reduces bus bandwidth usage and provides a foundation for efficient result merging on the software side, improving the response speed and system throughput of complex queries. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the database query hardware acceleration system architecture according to an embodiment of this application; Figure 2 This is a flowchart of the query plan fragment splitting and hardware unloading judgment in an embodiment of this application; Figure 3 This is a block diagram of the PCIe query acceleration hardware device module structure according to an embodiment of this application; Figure 4 This is a flowchart illustrating the hardware offloading revenue calculation process according to an embodiment of this application; Figure 5 This is a flowchart of the transaction snapshot consistency verification process according to an embodiment of this application; Figure 6 This is a flowchart illustrating the generation and transmission of intermediate compression results according to an embodiment of this application. Figure 7 This is a schematic diagram of the hardware query pipeline structure in an embodiment of this application.

[0018] Among them, 100-database server, 101-database executor, 102-host memory, 200-hardware acceleration device, 201-PCIe endpoint interface module, 202-DMA data transfer module, 203-query descriptor parsing module, 204-consistency snapshot verification module, 205-on-chip data rearrangement module, 206-hardware query pipeline, 207-compressed intermediate result generation module, 208-result return module, S101-S113-method step labels. Detailed Implementation

[0019] To make the objectives, technical solutions, and technical effects of this application clearer, the relevant technical solutions are further described in detail below with reference to the accompanying drawings in the embodiments of this application. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. Other implementation methods obtained by those skilled in the art based on the embodiments described in this application without creative effort should all fall within the protection scope of this application.

[0020] Unless otherwise defined, the technical and scientific terms used herein have the meanings commonly understood by one of ordinary skill in the art to which this application pertains. The related terms used in the specification are for illustrative purposes only and should not be construed as limiting the scope of protection of this application.

[0021] Example 1 like Figure 2 As shown, this embodiment provides a hardware acceleration method for database queries, which is applied to the database executor 101 in the database server 100. Using this method, the database executor 101 can dynamically evaluate the benefits of hardware acceleration based on the characteristics of the query task and the system operating status, and implement fine-grained hardware offloading while meeting transaction consistency requirements. Specifically, the method includes the following steps S101 to S105.

[0022] Step S101: Obtain a query plan for the target data and identify hardware-executable query segments from the query plan.

[0023] Specifically, after receiving an SQL query request, the database executor 101 first generates a query plan tree containing multiple operator nodes through the query optimizer. Then, it traverses this query plan tree and, based on a preset operator whitelist or feature matching rules, selects query segments suitable for execution by the hardware acceleration device 200. This segment-level identification mechanism provides a basis for targeted offloading, enabling the system to break down query tasks into hardware-accelerated and software-executed parts, rather than indiscriminately handing over entire SQL statements or entire data tables to hardware for processing. For example, for query plans containing complex nested loop joins or user-defined functions, the database executor 101 only identifies computationally intensive and logically structured scan, filter, or aggregation segments as hardware-executable query segments, while retaining the remaining segments unsuitable for hardware processing for software execution. This approach reduces the likelihood of overall rollback or execution anomalies caused by hardware not supporting specific operators, improving system compatibility and flexibility.

[0024] Step S102: Determine the data migration cost index and result compression benefit index corresponding to the query segment.

[0025] Specifically, after identifying a query segment that can be executed in hardware, the database executor 101 needs to quantitatively evaluate the offloading value of that segment. This embodiment introduces two independent and complementary evaluation dimensions: a data migration cost metric and a result compression benefit metric. The data migration cost metric characterizes the transmission overhead incurred in moving the data involved in the query segment from the host memory 102 to the hardware acceleration device 200, and in sending the processing results back to the host memory 102. The result compression benefit metric characterizes the degree of data reduction achieved by the hardware acceleration device 200 in processing the query segment by generating intermediate data in a non-complete result set. These two metrics together form the basis of joint decision-making and respectively reflect the transmission cost and the result compression benefit.

[0026] If only data migration costs are considered while compression benefits are ignored, acceleration opportunities may be missed in scenarios with low selectivity but high compression potential. Conversely, if only compression benefits are considered while migration costs are ignored, negative benefits may result due to excessive transmission time when bus congestion or large data volumes occur. Through the above-mentioned two-dimensional evaluation mechanism, this embodiment can improve the problem of insufficient evaluation results when judging solely based on data size or selectivity.

[0027] Step S103: Determine the hardware offloading evaluation value based on the data migration cost index and the result compression benefit index.

[0028] Specifically, the database executor 101 incorporates the two aforementioned metrics into a unified evaluation model for comprehensive consideration, calculating a hardware offload evaluation value. This hardware offload evaluation value characterizes the expected net benefit of offloading query fragments to hardware execution relative to pure software execution. In this embodiment, the determination of the hardware offload evaluation value is not based solely on a single fixed threshold, but rather considers a dynamic balance between transmission costs and compression gains.

[0029] For example, when data migration costs are high but compression benefits are significant, the hardware offloading evaluation value may still be positive, indicating that hardware offloading has the expected benefits. Conversely, when data migration costs are low but compression benefits are small, the hardware offloading evaluation value may be relatively low, indicating that the necessity of hardware offloading is weak. This comprehensive trade-off mechanism enables the system to trigger hardware acceleration in scenarios where performance benefits are expected, thereby reducing the additional resource consumption of low-benefit offloading.

[0030] It should be understood that the specific calculation method for the hardware offloading assessment value can be configured according to the actual system structure and operational requirements. For example, weighted summation, difference calculation, multi-factor decision tree, or other calculation methods that can characterize the combined impact of data migration costs and result compression benefits can be adopted.

[0031] In step S104, in response to the hardware unloading evaluation value meeting the preset conditions, an acceleration descriptor containing transaction version verification information is generated and the acceleration descriptor is sent to the hardware acceleration device.

[0032] Specifically, when the calculated hardware offload evaluation value meets the preset offload conditions, such as being greater than zero or exceeding a preset threshold, the database executor 101 determines that the query fragment is suitable for hardware execution and generates an acceleration descriptor accordingly. The acceleration descriptor is the information exchange carrier between the host side and the hardware side. It includes the operator type, data address, query conditions, and other operation instructions required for hardware execution, and encapsulates transaction version verification information.

[0033] Transaction version verification information is used to instruct the hardware acceleration device 200 to perform consistency verification on the read data blocks before executing a query, in order to determine whether the data version meets the requirements of the current database transaction view. For example, the transaction version verification information can be the data page version number or the log sequence number (LSN) corresponding to the current transaction. By writing the transaction version verification information as a field into the acceleration descriptor, the hardware acceleration device 200 can perform consistency verification based on this information with reduced host CPU intervention, thereby ensuring that the hardware offloading process meets the corresponding transaction consistency requirements.

[0034] After the acceleration descriptor is generated, the database executor 101 sends the acceleration descriptor to a designated register or shared memory region of the hardware acceleration device 200 via a peripheral interconnect bus, such as a PCIe bus, to trigger the hardware-side processing flow.

[0035] Step S105: Receive the intermediate compression result returned by the hardware acceleration device, and generate a query response based on the intermediate compression result.

[0036] Specifically, after completing the query processing, the hardware acceleration device 200 may not directly return the complete target data table, but instead return a compressed or compacted intermediate result. After receiving the compressed intermediate result, the database executor 101 parses or decodes it according to the contextual semantics of the query plan, and can merge it with the execution result of the software execution fragment to generate a complete query response and return it to the client.

[0037] Since the amount of data in the compressed intermediate results is usually smaller than that in the complete result set, this processing method helps reduce the bandwidth usage of the peripheral interconnect bus and shorten the data transmission time during query processing. Meanwhile, the compressed intermediate results are generated by the hardware acceleration device 200 after completing consistency verification based on transaction version verification information. Therefore, the database executor 101 can use the results if the verification passes, reducing the processing overhead caused by repeated version verification or data filtering on the software side.

[0038] Therefore, it is possible to meet data consistency requirements while also taking into account transmission efficiency and hardware acceleration benefits.

[0039] Example 2 like Figure 4 As shown, this embodiment, based on embodiment 1, further provides a quantitative calculation model for hardware offloading evaluation values. Specifically, it determines the data migration cost index based on data read transmission overhead and data return transmission overhead; and determines the result compression benefit index based on the estimated original result data volume and the estimated compressed result data volume.

[0040] In this embodiment, the data migration cost index can be determined not only by referring to empirical estimates but also by quantification based on actual physical transmission parameters. Specifically, the data read and transmission overhead refers to the time cost required to move the data block to be processed from the host memory 102 to the hardware acceleration device 200. This overhead can be calculated by querying the current effective bandwidth of the peripheral interconnect bus, the startup latency of the DMA data migration module 202, and the data block size. Alternatively, it can be obtained by sampling the time consumption of historical DMA transfer tasks using a sliding window to obtain the real-time average.

[0041] Data backhaul overhead refers to the time cost required to write the processing result from the hardware acceleration device 200 back to the host memory 102. Its calculation logic is similar to that of data read overhead, but the actual size of the backhaul result data also needs to be considered. By measuring the read overhead and backhaul overhead separately, the evaluation model can reflect operational conditions such as bus read / write asymmetry or unidirectional congestion.

[0042] For the result compression benefit metric, the estimated original result data volume is usually estimated by the query optimizer in the database executor 101 based on data table statistics, histograms, and predicate selectivity; the estimated compressed result data volume can be derived based on the operator type corresponding to the query fragment and the preset compressed intermediate result format, such as bitmap density or number of aggregation groups.

[0043] For example, for filtering operators, the estimated compressed data volume can be determined based on "number of rows in data blocks × bitmap compression ratio"; for aggregation operators, the estimated compressed data volume can be determined based on "estimated number of groups × size of a single aggregation record". Through this method, this embodiment transforms the abstract result compression benefit into a measurable difference in data volume, providing a data foundation for subsequent unified conversion to the time dimension.

[0044] Furthermore, to unify the aforementioned cost and benefit indicators into a time-based comprehensive decision-making process, this embodiment introduces a dynamic load factor. Specifically, it acquires the host processing load status and the hardware pipeline load status; based on the host processing load status and the hardware pipeline load status, it determines the computation time savings, relocation time increase, and compression time savings; based on the computation time savings, relocation time increase, and compression time savings, it calculates the hardware offload evaluation value using the following formula: in, The hardware unloading evaluation value, To save time in the calculation, Add time to the relocation. This saves time for the compression.

[0045] In the above formula, the parameters are not static constants, but variables that can change dynamically with the system's operating state. These dynamic variables are used to reflect the characteristics of the evaluation model as it adjusts with the system's state.

[0046] Specifically, calculation saves time This represents the computation time saved on the host side after offloading the query fragment to hardware. This calculation can be adjusted not only based on the difference between the estimated computational load of the query fragment and the hardware processing capabilities, but also considering the host's processing load status. When the host CPU is under high load, releasing host computing resources contributes relatively more to the overall system throughput; therefore, the computation time saved is significant. This increases accordingly, reflecting the dynamic changes in the opportunity cost of host computing resources. In other words, all other things being equal, the higher the host load, the more significant the computational savings from hardware offloading.

[0047] Relocation adds time This represents the additional data transfer time incurred due to the introduction of hardware acceleration. Its calculation basis can be the sum of data read transfer overhead and data return transfer overhead, and can be adjusted in conjunction with the hardware pipeline load status. When the hardware query pipeline 206 inside the hardware acceleration device 200 is under high load, newly arriving DMA transfer tasks or query processing tasks may need to queue, causing the actual processing time to exceed the transfer time determined based on the theoretical bus bandwidth. Therefore, the relocation time increases. It can be increased as the hardware pipeline load increases to reflect the additional time overhead caused by task queuing and hardware resource contention in the hardware offload evaluation value, thereby reducing the situation of continuing to centrally distribute tasks when hardware resources are busy.

[0048] Compression saves time This represents the transmission time saved by returning compressed intermediate results instead of the complete result set. Its value can be calculated based on the difference between the estimated original result data volume and the estimated compressed result data volume, combined with the available bandwidth of the current peripheral interconnect bus. When the available bandwidth of the peripheral interconnect bus is low or bandwidth utilization is high, the same data reduction can usually save more transmission time. Therefore, the evaluation model can increase the offloading priority of high-compression-ratio query segments in scenarios where bus resources are relatively scarce, in order to adaptively allocate transmission resources.

[0049] It should be understood that the above formula is only an exemplary linear calculation model. In other embodiments, nonlinear functions, piecewise thresholds, multi-factor decision trees, or other forms can also be used to express the trade-offs between the time components, as long as their core logic covers the dynamic trade-offs between the three dimensions of time saving in computation, time increase from relocation, and time saving in compression, they can all be used to implement the technical solution described in this application.

[0050] The dynamic quantification model described above can reduce the bias caused by relying solely on fixed rules for unloading judgments, and make the technical solution feasible to adapt to different system loads, data distributions, and bus states.

[0051] Example 3 like Figure 6As shown, this embodiment, based on Embodiment 1, further details the generation mechanism and specific form of the compressed intermediate results. As one implementation, the acceleration descriptor also includes result return format indication information; the compressed intermediate results are intermediate data of an incomplete result set; receiving the compressed intermediate results returned by the hardware acceleration device, including receiving the intermediate data generated by the hardware acceleration device according to the result return format indication information.

[0052] Specifically, the result return format indication information is control information sent by the database executor 101 to the hardware acceleration device 200, used to specify the data structure used when the hardware encapsulates the processing results. Through this result return format indication information, the hardware output format can be configured according to the semantic requirements of the upper-layer query plan, and is not limited to outputting fixed format data or a complete result table.

[0053] In this embodiment, compressed intermediate results are defined as intermediate data of an incomplete result set. That is, the data returned by the hardware acceleration device 200 is not the complete target data table finally presented to the user, but intermediate data that has been processed by specific operators and still needs to be further merged, decoded, or materialized by the software side. For example, for aggregation queries, the hardware acceleration device 200 returns the local cumulative values ​​of each group, rather than the final statistical results; for filtering queries, the hardware acceleration device 200 returns row-level matching tags, rather than the complete row records that meet the conditions.

[0054] By reducing redundant fields, aligning padding, and carrying complete row records, the aforementioned intermediate data can improve transmission compactness while ensuring the semantics of the query are recoverable, and further reduce the bandwidth usage of the peripheral interconnect bus.

[0055] In one optional implementation, the result return format indication information may further include data structure type, field arrangement order, result buffer address, valid data length, and verification information, so that the host side can parse the corresponding intermediate results according to the agreed format.

[0056] Furthermore, to illustrate the specific implementation of the aforementioned incomplete result set intermediate data, this embodiment lists four typical intermediate data formats to adapt to different types of query operators. As one implementation, the intermediate data includes at least one of the following: when the query segment is a filtering query, the intermediate data is a filtering bitmap; when the query segment is an aggregation query, the intermediate data is an aggregation intermediate value; when the query segment is a join query, the intermediate data is a join matching index; when the query segment is a sorting query, the intermediate data is a sorting index or a Top-N candidate set.

[0057] Specifically, when the query segment is a filtering query, the filtering bitmap generated by the hardware acceleration device 200 is a compact data structure formed using bitwise encoding. Compared to returning complete row records or Boolean vector arrays that meet the conditions, the filtering bitmap uses bit operations to represent the filtering result of each row as a single bit.

[0058] For example, for a data block containing 1 million rows of data, under the example condition of a 50% filter selectivity, when a single row contains multiple fields, returning the complete row record may require tens to hundreds of MB of data transfer, while returning the uncompressed filter bitmap requires approximately 122KB. This difference in data volume stems from the fact that the bitmap format reduces the carrying of field data, field alignment padding, and repetitive structural information, primarily preserving the correspondence between row positions and the filtering results. After receiving the filter bitmap, the database executor 101 can use it as an input mask for subsequent operators or extract the offset of valid rows through bitwise operations, thereby completing the transmission of the filtering results with lower transmission overhead.

[0059] When the query segment is an aggregate query, the intermediate aggregate value generated by the hardware acceleration device 200 adopts a local aggregation processing method. In a distributed or hierarchical computing architecture, the complete aggregation result usually needs to be obtained through global merging. In this embodiment, the hardware can perform local aggregation operations on the current data block to generate intermediate results containing grouping keys and local cumulative values, instead of completing global aggregation across all data blocks in a single hardware processing task.

[0060] For example, when performing a SUM or COUNT operation, the hardware acceleration device 200 returns the partial sum and / or partial count of each packet within the current data block; when performing an AVG operation, it returns a combination of the partial sum and the partial count. This processing method reduces the possibility of aggregation result deviations due to a lack of global data distribution information, and also changes the amount of returned data from primarily increasing with the original row count to primarily increasing with the packet base, thereby reducing the bus bandwidth occupied by data return.

[0061] When the query segment is a join query, the join matching index generated by the hardware acceleration device 200 is used to characterize the association between data blocks, rather than directly materializing the wide table result after the join. In specific implementations, this join matching index can be a hash digest table or a list of row identifiers.

[0062] For example, in a hash join scenario, the hardware can generate a hash match digest for the probe end, or simply record a list of RowIDs for successfully matched Build ends and send this list back to the host. Compared to sending back a complete result row containing all join fields, sending back only the index or digest typically reduces the amount of data sent back, especially when there are many join fields or a wide matching result row. The database executor 101 can then use the corresponding index to locate and assemble the final result on the software side, or pass the corresponding index as a filtering condition for a half-join to downstream operators.

[0063] The above strategy can be used for star schema queries or join queries with high filtering selectivity, which helps reduce the possibility of increased PCIe bus load due to the expansion of join results.

[0064] When the query segment is a sorted query, the sorted index or Top-N candidate set generated by the hardware acceleration device 200 can adopt a heap structure or an ordered pointer mechanism. For full sorting, the hardware acceleration device 200 may not return the complete sorted records, but instead return an ordered index or pointer array pointing to the original data positions; for Top-N queries, the hardware acceleration device 200 can maintain a min-heap or max-heap of size N on-chip, and return N candidate values ​​and their corresponding row identifiers after processing the current data block.

[0065] For example, when querying the top 100 orders with the highest amounts, regardless of the number of records in the input data block, the hardware output for a single data block can be limited to 100 candidate records or 100 index entries. This fixed-length or limited-length output method reduces the direct impact of input data size on output transmission overhead and weakens the limitation of result set size on the hardware acceleration benefits of sorting queries, thus helping to improve system stability and response efficiency in high-concurrency sorting scenarios.

[0066] It should be understood that the four intermediate data types mentioned above are for illustrative purposes only and are not intended to exhaustively limit the possible data formats. In practical applications, depending on the semantics of the query operator, the data organization format, and the hardware processing capabilities, projected columnar storage fragments, intermediate vectors from expression calculations, or other incomplete result sets of intermediate data can be further employed. As long as the corresponding data is compressed, indexed, or abstracted to replace the direct transmission of the complete result table, and can be generated on demand according to the result return format indication information, it can be used to implement the technical solution described in this application.

[0067] Example 4 like Figure 5As shown, this embodiment, based on Embodiment 1, further details the mechanism for generating transaction version verification information. As one implementation method, a transaction snapshot identifier is dynamically generated based on the current transaction view; the transaction snapshot identifier is encapsulated as transaction version verification information into an acceleration descriptor; wherein, the transaction snapshot identifier includes at least one of a data page version number or a log sequence number.

[0068] Specifically, when the database executor 101 decides to unload a query fragment to the hardware acceleration device 200, it can obtain the transaction context state corresponding to the current query in real time and dynamically generate a transaction snapshot identifier accordingly, instead of directly using pre-configured fixed version parameters. This dynamic generation process occurs, for example, when generating the acceleration descriptor in step S104.

[0069] The database executor 101 first obtains the transaction view of the current transaction, which is used to limit the range of data versions visible to the current transaction; then, it extracts identifiers that can characterize the timeliness of the data from the transaction view or related version metadata, such as data page version number or log sequence number.

[0070] A version verification field (Version_Tag Field) can be set in the data structure of the acceleration descriptor. The database executor 101 writes the extracted identifier into the version verification field, so that the version information is issued together with the current hardware operation command. Thus, the hardware acceleration device 200 can perform data consistency verification based on the version benchmark corresponding to the current acceleration task, so that the verification basis used on the hardware side corresponds to the transaction view when the host side initiates the query.

[0071] The dynamic generation method is adopted because in a high-concurrency database environment, the data state may continuously change as concurrent transactions are executed. If a globally fixed version identifier is used, or a version identifier cached at the start of the session and not updated, when the underlying data pages are updated and committed by other concurrent transactions during query execution, the hardware acceleration device 200 may read data versions that are outside the visibility range of the current transaction or do not meet the current read requirements, thus affecting the consistency between the query results and the transaction isolation requirements.

[0072] By obtaining the transaction snapshot identifier corresponding to the current task from the current transaction view when generating each acceleration descriptor, each hardware unloading task can be bound to the data version baseline corresponding to the point of generation of the task, thereby reducing the possibility of data consistency deviations caused by version changes.

[0073] The specific form and verification logic of transaction snapshot identifiers can differ depending on the database storage engine and concurrency control mechanism. This embodiment provides two typical implementation methods for illustration.

[0074] In databases employing Multi-Version Concurrency Control (MVCC), the transaction snapshot identifier can be a transaction ID or version watermark information from a read view. Database executor 101 can write the current transaction's start timestamp, transaction ID, or relevant parameters from the read view used to determine version visibility into the acceleration descriptor.

[0075] After reading the data block, the hardware acceleration device 200 can parse the version information stored in the data page or row record, such as the transaction ID xmin for creating the row record and the transaction ID xmax for deleting the row record, and compare them with the transaction snapshot identifier in the acceleration descriptor to determine whether the corresponding row record is visible to the current transaction. For row records that do not meet the visibility rules of the current transaction, the hardware query pipeline 206 can skip processing or return the corresponding abnormal status to the host side.

[0076] In databases employing Write-Ahead Logging (WAL) or log-based recovery mechanisms, the transaction snapshot identifier can be the Log Sequence Number (LSN). The LSN is an identifier in the log that changes according to the writing order and can be used to characterize the log position corresponding to the database state. The database executor 101 can write the LSN that has been confirmed as readable for the current transaction, or the LSN corresponding to the current consistent view, into the acceleration descriptor. After reading a data page, the hardware acceleration device 200 can check the PageLSN recorded in the data page header and compare the PageLSN with the LSN in the acceleration descriptor to determine whether the data page meets the data version requirements corresponding to the current query task.

[0077] It should be understood that the MVCC and WAL described above are merely examples. In other implementations, the transaction snapshot identifier can also be a hybrid logical clock, a vector clock, or other metadata that can characterize the data version order and causal relationship. As long as the corresponding metadata can be used by the hardware side to verify the consistency relationship between the data block and the query transaction, it can be used to implement the technical solution described in this application.

[0078] This embodiment enables the hardware acceleration device 200 to perform data consistency checks with less reliance on page-by-page verification by dynamically generating and distributing transaction snapshot identifiers. This approach reduces the processing overhead of verifying a large number of data pages one by one on the host side before unloading, and avoids the hardware from continuing to process data that does not meet the visibility requirements of the current transaction, thereby reducing invalid calculations and erroneous result backhauls. Under the premise of meeting the database transaction isolation and consistency requirements, the above method is beneficial to improving the execution efficiency of fine-grained hardware unloading.

[0079] Example 5 like Figure 2 As shown, this embodiment, based on Embodiment 1, further details the identification scope of query plan fragments and the collaborative merging mechanism of software and hardware execution results. As one implementation method, query fragments containing at least one of the following operators are identified from the query plan as hardware-executable query fragments: data block scan operator, predicate filtering operator, field projection operator, expression calculation operator, hash aggregation operator, semi-join operator, hash join operator, Top-N filtering operator, and local sorting operator.

[0080] Specifically, the above list of operators is used to illustrate the scope of unloadable operators and does not constitute an exhaustive limitation on the types of unloadable operators. In practical applications, if the operators in the query plan have high data parallelism, few control flow branches, and relatively regular computational logic, and can be mapped to pipelined operations within the hardware acceleration device 200, then they can be included in the set of unloadable operators.

[0081] For example, the data block scan operator and the predicate filter operator have Single Instruction Multiple Data (SIMD) processing characteristics, and are generally suitable for vectorized processing by the hardware query pipeline 206; although the hash aggregation operator and the hash join operator involve random memory access, they can be combined with on-chip hash tables, Bloom filters or other hardware data structures to improve processing throughput; the Top-N filtering operator and the local sorting operator can use heap structures or sorting networks to implement fixed-length or limited-length data processing.

[0082] With the development of hardware architecture and query types, vector retrieval operators, graph traversal operators, or other operators that can be executed by hardware pipelines can also be included in the unloadable scope based on the same fragment-level unloading principle, so as to improve the adaptability of the unloading mechanism to different query scenarios.

[0083] While identifying hardware-executable query segments, this embodiment also retains software execution segments in the query plan other than hardware-executable query segments. This processing method forms a hardware-software co-execution structure. The hardware acceleration device 200 has high processing throughput when processing operators with high data parallelism and relatively regular computational logic, but its execution flexibility is generally lower than that of general-purpose processors.

[0084] For operators that contain complex control flow, irregular memory access patterns, or rely on external library functions, such as nested loop joins, user-defined functions (UDFs), complex XML / JSON parsing, or recursive queries, directly offloading them to hardware may not yield the expected performance gains, and may also cause execution exceptions due to hardware incompatibility with the corresponding operations.

[0085] Therefore, when splitting the query plan, the database executor 101 can retain operators unsuitable for hardware processing as software execution fragments, which will continue to be executed by the host-side CPU. This hybrid execution model enables the system to utilize hardware acceleration for computationally intensive tasks while retaining the ability of software to execute complex logic, thereby improving overall processing performance while maintaining the semantic integrity of the query.

[0086] Furthermore, to integrate the outputs of the hardware acceleration segment and the software execution segment into a unified query result, this embodiment further explains the result merging method. As one implementation method, the compressed intermediate result is merged with the execution result of the software execution segment to generate a query response.

[0087] The "merging" here does not simply refer to data splicing, but rather to the corresponding combination processing based on the topological relationship of the query plan, operator semantics, and the type of compressed intermediate results. Since the hardware acceleration device 200 returns intermediate data that is not a complete result set, as described in Embodiment 3, the database executor 101 can adopt corresponding parsing, merging, index positioning, or result materialization strategies for different operator types, thereby forming a final result that conforms to the query semantics.

[0088] Specifically, for different types of compressed intermediate results, the merging operation can include the following typical scenarios.

[0089] When the intermediate compression result returned by the hardware acceleration device 200 is a filter bitmap, the database executor 101 can use the filter bitmap as an input mask for subsequent software execution segments. For example, if the hardware is responsible for scanning and filtering in the query plan, and the software is responsible for subsequent projection and joining, the database executor 101 can directly pass the filter bitmap to the projection operator or join operator on the software side instead of materializing the complete row records according to the filter bitmap first, so that the corresponding operator only accesses the rows marked as valid in the filter bitmap. The above method can reduce the repeated copying of intermediate data, memory allocation, and the advance materialization of complete row records.

[0090] When the intermediate compression result returned by the hardware acceleration device 200 is an aggregated intermediate value, the database executor 101 can perform a global merge operation on the software side. For example, the hardware acceleration device 200 calculates local SUM values ​​and local COUNT values ​​for multiple data blocks respectively, and the software side can accumulate each local value according to the grouping key to obtain the corresponding global aggregation result; for AVG operations, each local part sum and part count can be accumulated separately, and the average value can be calculated based on the accumulated sum and the total count.

[0091] The two-stage aggregation method described above can delegate the extensive traversal and local computation of the original data to the hardware side, allowing the software side to mainly process the relatively small amount of intermediate state data, thereby reducing the computational load on the host CPU.

[0092] When the compressed intermediate result returned by the hardware acceleration device 200 is a join matching index or a sorting index, the database executor 101 can use the corresponding index to locate the original data on the software side and complete the final result assembly or sorting. For example, in a semi-join scenario, the RowID list returned by the hardware on the Build side can be used as a filtering condition on the Probe side, and the software side can read the matching rows and perform subsequent join processing accordingly; in a Top-N scenario, the candidate set returned by the hardware acceleration device 200 can include N candidate records or the row identifier and sorting key of the corresponding records, and the software side can further perform candidate set merging, validation, global sorting, or formatted output.

[0093] Through the above-mentioned multiple merging methods, when data is transmitted back on the hardware side using different compact formats, the software side can process the intermediate results according to the corresponding operator semantics and transform them into the final query response that conforms to the SQL query semantics, thereby completing the processing flow from unloading query fragments to merging the final results.

[0094] Example 6 like Figure 3 and Figure 5As shown, this embodiment provides a hardware acceleration method for database queries, which is applied to a hardware acceleration device 200. Unlike the previous embodiments, which mainly described host-side offloading decisions, this embodiment mainly describes the execution logic inside the hardware acceleration device 200, forming a relatively independent hardware-side processing flow. Through this implementation, the coordination relationship between each processing step and functional module on the hardware acceleration device side can be independently explained. Specifically, the method includes the following steps S201 to S205.

[0095] Step S201: Receive the acceleration descriptor sent by the host. The acceleration descriptor contains transaction version verification information and result return format indication information.

[0096] Specifically, the hardware acceleration device 200 receives write requests on the peripheral interconnect bus through the PCIe endpoint interface module 201. When a descriptor write signal sent to this device is detected, the corresponding acceleration descriptor can be latched into the internal command queue. The query descriptor parsing module 203 then parses the acceleration descriptor and extracts the control fields required for subsequent processing.

[0097] The transaction version verification information indicates the data version baseline corresponding to the current query task and serves as the basis for consistency verification on the hardware side. The result return format indication information specifies the encapsulation form of the hardware output results, such as a filtered bitmap, aggregated intermediate values, or join matching indexes. During this stage, the hardware acceleration device 200 primarily receives and parses the acceleration descriptors, without participating in the host-side evaluation of whether to perform hardware offloading, thus achieving functional separation between hardware execution logic and host scheduling logic.

[0098] Step S202: Obtain data blocks based on accelerated descriptors through the direct memory access channel.

[0099] Specifically, after parsing and obtaining the physical address or logical offset of the data block, the DMA data transfer module 202 initiates a direct memory access request, reads the target data block directly from the host memory 102 via the peripheral interconnect bus, and writes the target data block into the on-chip cache or local storage of the hardware acceleration device 200. The above data transfer process can be primarily completed by hardware, without requiring the host CPU to intervene in the sequential data copying and transfer operations, thereby reducing the host CPU's involvement in the data transfer process.

[0100] For example, the DMA data transfer module 202 can automatically split read transactions, manage bus requests, and execute corresponding data receiving operations according to the data block size and starting address specified in the acceleration descriptor, thereby improving the utilization rate of the effective bandwidth of the peripheral interconnect bus. This descriptor-driven DMA mechanism enables the hardware acceleration device 200 to acquire data to be processed with higher transmission efficiency and reduces the transmission latency and computational resource consumption caused by CPU involvement in data copying.

[0101] Step S203: Perform consistency verification on the data blocks based on the transaction version verification information.

[0102] Specifically, after a data block is read into the hardware acceleration device 200, it can first undergo version verification by the consistency snapshot verification module 204, instead of directly entering the hardware query pipeline 206. The consistency snapshot verification module 204 extracts the actual version identifier of the data block, such as the page version number or LSN, from the data block header or the corresponding metadata area, and compares it with the transaction version verification information carried in the acceleration descriptor.

[0103] When both satisfy the preset consistency rules, such as the data block version being no later than the version allowed by the transaction snapshot, the consistency check can be considered successful. The preset consistency rules can be configured according to the transaction isolation level adopted by the database, the version metadata organization method, and the storage engine implementation method, and the corresponding verification parameters can be provided to the consistency snapshot verification module 204 by the query descriptor parsing module 203.

[0104] This step is the hardware-side processing stage to ensure database transaction consistency. Since the hardware acceleration device 200 can read data via direct memory access, without corresponding version verification, it might read data versions that are not yet visible to the current query or no longer meet the current read requirements. By setting up version comparison logic within the hardware, the data entering the subsequent query pipeline can meet the transaction view requirements corresponding to the task issued by the host side, thereby maintaining hardware processing throughput while meeting data correctness requirements.

[0105] In step S204, in response to the successful consistency check, hardware query processing is performed on the data block to generate a compressed intermediate result of an incomplete result set.

[0106] Specifically, once the consistency snapshot verification module 204 confirms that the data block version meets the requirements, the data is sent to the hardware query pipeline 206 for query operator processing. The hardware query pipeline 206 configures the corresponding processing unit according to the operator type specified in the acceleration descriptor, such as filtering, aggregation, or joining, and performs the corresponding calculation operation on the data stream.

[0107] In this embodiment, the hardware query pipeline 206 is configured to generate compressed intermediate results of incomplete result sets. That is, the hardware side completes the local computation tasks defined by the query plan fragment and represents the processing results as compact intermediate data, rather than directly outputting the complete target data table.

[0108] For example, for aggregate queries, hardware query pipeline 206 can output local cumulative values ​​corresponding to each group, rather than the final statistical result formed across all data blocks; for filter queries, hardware query pipeline 206 can output a filter bitmap representing the row matching status, rather than a complete row record containing multiple fields.

[0109] The above design enables the hardware acceleration device 200 to output intermediate results that can be further processed by the database executor 101, allowing the hardware acceleration device 200 to participate in query execution as a coprocessor node in the query plan, thereby reducing the occurrence of calculation deviations or redundant transmissions due to the lack of a global query context on the hardware side.

[0110] Step S205: According to the result return format instruction information, the compressed intermediate result is returned to the host.

[0111] Specifically, after the hardware query pipeline 206 completes the query processing and generates compressed intermediate results, the compressed intermediate results generation module 207 can encapsulate the calculation results into a corresponding compact data structure according to the result return format indication information in the acceleration descriptor. The result return module 208 then writes the compressed intermediate results back to the pre-specified result buffer in the host memory 102 through the PCIe endpoint interface module 201.

[0112] Because the returned results are represented in a compact intermediate state, their data size is typically smaller than the original data block or the complete result set, thus helping to reduce the bandwidth usage of the peripheral interconnect bus. For example, when processing filtering tasks with tens of millions of records, the data size of the returned filter bitmap can be several MB, while the returned complete result set containing multiple fields can reach GB levels, depending on the record width, filter selectivity, and result format.

[0113] The aforementioned on-demand formatted return mechanism can shorten data return time and reserve corresponding bus resources for other concurrent tasks, thereby improving the overall throughput and concurrent processing capability of the database system. Although this embodiment uses the PCIe interface as an example, in other embodiments, CXL, NVLink, or other high-speed interconnect protocols that support direct memory access can also be used to realize data interaction between the host and the hardware acceleration device 200.

[0114] Example 7 like Figure 5 and Figure 7As shown, this embodiment, based on embodiment 6, further details the exception handling mechanism and data adaptation process within the hardware acceleration device 200. As one implementation method, in response to a failure to pass the consistency check, a reread instruction or a software execution instruction is returned to the host.

[0115] Specifically, when the consistency snapshot verification module 204 determines in step S203 that the version of the currently read data block does not match the transaction version verification information in the acceleration descriptor, the hardware acceleration device 200 can stop performing subsequent calculations on that data block, not return intermediate results generated based on that data block, and trigger an exception handling process. This processing method avoids results generated based on inconsistent version data from entering the final query response, thus satisfying the database transaction isolation and result correctness requirements.

[0116] In one embodiment, the hardware acceleration device 200 returns control commands to the database server 100 through the PCIe endpoint interface module 201. The control commands include at least one of a reread command or a software execution command.

[0117] When a reread instruction is returned, it can carry the current actual version number of the data block, the log sequence number, or the corresponding exception status code. After receiving the reread instruction, the database server 100 can update the version metadata in its local cache, reload the compliant data block from the corresponding storage area, or regenerate the acceleration descriptor containing the corresponding transaction version verification information and reissue it. Thus, the hardware can provide version deviation information to the host side, forming a hardware-software collaborative version deviation correction mechanism.

[0118] When the system returns to software execution, it can indicate that the version conflict of the current data block is complex or recurring, or that the hardware pipeline is not suitable to continue waiting. In this case, the hardware acceleration device 200 can release the occupied computing resources and command queue slots, and notify the database server 100 to transfer the corresponding query segment to software execution. This anomaly feedback mechanism can reduce the situation where hardware occupies queues or computing resources for extended periods while waiting for data version updates, and helps improve system stability and resource utilization in concurrent update scenarios.

[0119] As another implementation, hardware query processing is performed on the data block, including: on-chip data rearrangement of the data block to convert the data block into a data stream adapted to the hardware pipeline; and performing query processing on the data stream through the hardware query pipeline 206.

[0120] Specifically, even if the consistency check passes, data blocks read from host memory 102 should generally not be directly input into the hardware query pipeline 206 without conversion. The underlying storage format of the database is usually organized for disk I / O efficiency, cache utilization, or compression ratio, such as using row-based storage, column-based storage, or page-level compression. The data page may also include page header metadata, row offset arrays, null bitmaps, and variable-length fields.

[0121] Hardware query pipeline 206 is typically designed for high-throughput parallel computing, and its input data can be fixed-length, memory-aligned, and contiguously arranged data streams. Database storage formats and hardware computing formats may differ in data semantics and physical layout. If raw data pages are directly fed into the computing unit, some clock cycles may be used to parse page headers, skip irrelevant bytes, or handle unaligned accesses, thereby reducing the effective utilization of hardware query pipeline 206.

[0122] Therefore, this embodiment includes an on-chip data rearrangement module 205 to achieve conversion between storage format and computation format. After the consistency check passes, the on-chip data rearrangement module 205 parses the physical layout of the data block, removes storage metadata irrelevant to the current query, and extracts the fields to be used in the computation according to the operator requirements specified in the acceleration descriptor. Subsequently, it performs decompression, type conversion, alignment padding, or sequential reorganization on the corresponding fields to form a fixed-length vector or structured data stream that meets the input requirements of the hardware query pipeline 206.

[0123] For example, for order table data blocks in row storage format, the on-chip data rearrangement module 205 can extract the "amount" field and "user ID" field scattered in each row and organize them into a continuous column vector; for data blocks stored using dictionary compression, the on-chip data rearrangement module 205 can complete the corresponding dictionary decoding on-chip and output the recovered numerical stream or encoded value stream.

[0124] The on-chip data rearrangement process described above reduces pipeline stalls caused by data format mismatches and improves the effective processing throughput of the hardware query pipeline 206. The on-chip data rearrangement logic can be configured according to the data format used by the database storage engine. Under different storage formats, the on-chip data rearrangement module 205 can be configured with corresponding field positioning, decompression, type conversion, and alignment parameters, and can reduce irrelevant data entering the hardware query pipeline 206 based on the fields required by the query segment.

[0125] Example 8 like Figure 1 As shown, this embodiment provides a database query hardware acceleration system. The system includes a database server 100 and a hardware acceleration device 200, which are connected via a peripheral interconnect bus.

[0126] Specifically, the database server 100 serves as the system's main control node and data processing node, and internally includes a database executor 101 and host memory 102. The database executor 101 is used to parse SQL queries, generate query plans, perform unloading evaluations, and merge final results; the host memory 102 is used to cache database data blocks, store acceleration descriptors, and receive compressed intermediate results.

[0127] The hardware acceleration device 200, acting as a dedicated coprocessor node, establishes a data path with the database server 100 via a peripheral interconnect bus. In this embodiment, the peripheral interconnect bus is preferably a PCIe (Peripheral Component Interconnect Express) bus, which utilizes its support for high-speed data transmission and direct memory access to enable instruction interaction and data transmission between the database server 100 and the hardware acceleration device 200.

[0128] It should be understood that although this embodiment uses the PCIe bus as an example for illustration, in other embodiments, the peripheral interconnect bus can also adopt CXL (Compute Express Link), NVLink, CCIX or other high-speed interconnect protocols that support cache coherency or direct memory access. As long as it can realize instruction issuance and data transfer between the database server 100 and the hardware acceleration device 200, it can be used to implement the technical solution described in this application.

[0129] Furthermore, the physical form of the hardware acceleration device 200 is not limited to a standalone PCIe expansion card; it can also be a dedicated chip integrated on the motherboard, an acceleration module connected via a CXL memory expander, or a shared acceleration unit in a rack-level resource pooling architecture.

[0130] The database server 100 is configured to execute the database query hardware acceleration method described in any one of Embodiments 1 to 5. In addition to having database management functions, the database server 100 may also be provided with software modules or firmware logic for identifying hardware-executable query segments, calculating data migration cost indicators and result compression benefit indicators, determining hardware offloading evaluation values, dynamically generating acceleration descriptors containing transaction version verification information, and processing intermediate compression results.

[0131] By acquiring the load status, executable operator type, available processing resources, and query task characteristics of the hardware acceleration device 200, the database server 100 can determine the unloading method appropriate to the current system operating state, rather than simply performing hardware unloading based on a fixed data scale or external scheduling instructions.

[0132] The hardware acceleration device 200 is configured to execute the database query hardware acceleration method described in Embodiment 6 or Embodiment 7. Specifically, the hardware acceleration device 200 internally includes a PCIe endpoint interface module 201, a DMA data transfer module 202, a query descriptor parsing module 203, a consistency snapshot verification module 204, an on-chip data rearrangement module 205, a hardware query pipeline 206, a compressed intermediate result generation module 207, and a result return module 208.

[0133] The PCIe endpoint interface module 201 is used to establish a bus connection between the hardware acceleration device 200 and the database server 100, and to receive acceleration descriptors or return corresponding result data. The DMA data transfer module 202 is used to retrieve data blocks from the host memory 102 via direct memory access according to the data address recorded in the acceleration descriptor, or to write the processing results back to the host memory 102.

[0134] The query descriptor parsing module 203 is used to parse the acceleration descriptor and extract fields such as operator type, data address, query conditions, transaction version verification information, and result return format indication information. The consistency snapshot verification module 204 is used to perform version consistency verification on the data blocks obtained by the DMA data migration module 202 based on the transaction version verification information.

[0135] The on-chip data rearrangement module 205 converts data blocks in the database storage format into a data stream adapted to the hardware query pipeline 206. The hardware query pipeline 206 performs scans, filters, aggregations, joins, sorts, or other query processing based on the operator type. The compressed intermediate result generation module 207 generates compressed intermediate results such as filter bitmaps, aggregated intermediate values, join matching indexes, sorting indexes, or Top-N candidate sets based on the query processing results and result return format instructions. The result return module 208 writes the compressed intermediate results back to a specified buffer in the host memory 102 and notifies the database server 100 via interrupts, status registers, or polling.

[0136] The aforementioned modules work together to form a hardware processing chain that includes receiving accelerated descriptors, performing DMA data reads, checking transaction consistency, implementing on-chip data rearrangement, performing pipelined polling, and generating and sending back compressed intermediate results. The corresponding functions in the hardware acceleration device 200 can be implemented using application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLAs), or other hardware logic structures to improve data processing throughput while meeting transaction consistency requirements.

[0137] In this system, the database server 100 and the hardware acceleration device 200 can work together according to a predefined interaction protocol. The database server 100 writes the generated acceleration descriptor into the command register or shared memory area of ​​the hardware acceleration device 200; after parsing the acceleration descriptor, the hardware acceleration device 200 reads the data block to be processed from the host memory 102 through the DMA data transfer module 202, so as to reduce the CPU's involvement in data copying.

[0138] After the hardware acceleration device 200 completes processing, it writes the compressed intermediate results back to the designated result buffer in the host memory 102 via the peripheral interconnect bus, and sends a completion notification to the database server 100 via an interrupt or polling mechanism. Upon receiving the completion notification, the database server 100 retrieves the compressed intermediate results and continues to execute software processing or result merging according to the query plan.

[0139] The aforementioned interaction method based on accelerated descriptor-driven and DMA data transfer reduces the processing overhead caused by the host CPU's involvement in data movement and enables the hardware acceleration device 200 to perform corresponding parallel query processing. Through this system architecture, the host-side offloading decision function and the hardware-side data processing function cooperate to form a processing flow that balances transmission efficiency, transaction consistency, and computational performance, and helps alleviate the performance and consistency problems caused by the separation of software and hardware processing flows.

[0140] Example 9 To further illustrate the implementation process of the technical solution of this application in a specific business scenario, the following example uses order data statistical query during high concurrency in e-commerce business. In this scenario, the database server 100 needs to perform an analysis query on an order table containing a large number of records. The query semantics are, for example, "Grouped by user identifier, statistically analyze the total amount of orders that meet a specific time range and whose order amount is greater than a preset threshold."

[0141] This query is characterized by a large input data scale, a low filtering selection rate, and a high computational load, and can serve as a typical application example to illustrate the hardware acceleration method of this application.

[0142] Specifically, after receiving a query request, the database executor 101 first generates a query plan and identifies data block scan operators, predicate filtering operators, and hash aggregation operators from the query plan, using them as hardware-executable query segments. Simultaneously, the database executor 101 retains software execution segments from the query plan that involve complex business logic validation, for later merging with the hardware-accelerated results.

[0143] The aforementioned fragment-level recognition method offloads the computational load that is more suitable for hardware parallel processing, while retaining the processing path for software to execute complex logic, thereby reducing the possibility of the entire query plan being rolled back due to hardware not supporting specific functions.

[0144] When determining the unloading strategy, the database executor 101 calculates the hardware unloading evaluation value based on the current system state and query characteristics. In this scenario, the amount of order data to be scanned reaches, for example, tens of gigabytes, resulting in relatively high data reading and transmission overhead. However, according to the query optimizer's estimation, records that meet the time range and amount threshold conditions account for, for example, about one-thousandth of the total number of records, and the size of the aggregated result set is significantly smaller than the original data volume. Therefore, the result compression benefit index is relatively high.

[0145] Meanwhile, the database executor 101 detects that the current host CPU is under high load, while the hardware query pipeline 206 of the hardware acceleration device 200 has a relatively low load. According to the evaluation model described in Example 2, although the relocation time T_add_trans increases due to the large amount of data, the computation time saving T_save_comp corresponding to the high host load and the compression time saving T_save_compress corresponding to the reduction of the result data volume are relatively large. Finally, the calculated hardware offload evaluation value V is higher than the preset threshold.

[0146] The dynamic evaluation process described above can identify query tasks with large input data volume and high potential for result compression, rather than abandoning uninstallation simply because the data volume is large, or performing uninstallation simply because the data volume is small.

[0147] In response to the hardware unloading evaluation value meeting preset conditions, the database executor 101 generates an acceleration descriptor. This acceleration descriptor includes operation information such as the starting address of the order data block, the data block size, time range filtering conditions, amount threshold filtering conditions, and user identifier aggregation key, as well as the log sequence number corresponding to the current transaction view, which serves as transaction version verification information.

[0148] In addition, the acceleration descriptor also includes result return format indication information, which instructs the hardware acceleration device 200 to generate and return aggregated intermediate values. The database executor 101 then sends the acceleration descriptor to the hardware acceleration device 200 via the PCIe bus.

[0149] After receiving the acceleration descriptor, the hardware acceleration device 200 queries the descriptor parsing module 203 to parse the acceleration descriptor and provides the DMA data transfer module 202 with the starting address and size of the order data block. The DMA data transfer module 202 reads the order data block from the host memory 102 according to the corresponding address information to reduce the host CPU's involvement in data copying.

[0150] Before the order data block enters the hardware query pipeline 206, the consistency snapshot verification module 204 extracts the PageLSN from the header of the order data block and compares the PageLSN with the transaction LSN carried in the acceleration descriptor to verify whether the data block meets the transaction consistency requirements corresponding to the current query task.

[0151] When the consistency check passes, the on-chip data rearrangement module 205 parses the storage format of the order data block, extracts the order time field, order amount field and user identifier field, and converts the above fields into a continuous data stream adapted to the hardware query pipeline 206.

[0152] Hardware query pipeline 206 sequentially performs time range filtering, amount threshold filtering, and local aggregation based on user identifier on the data stream. For order records that meet the time range and amount threshold conditions, hardware query pipeline 206 adds the order amount to the corresponding local aggregation state according to the user identifier.

[0153] After the hardware query pipeline 206 completes the processing of the current data block, the compressed intermediate result generation module 207 generates an aggregated intermediate value list including the user identifier and the corresponding local accumulated amount. The hardware acceleration device 200 may not send back all filtered order detail records, but instead send back the above-mentioned aggregated intermediate value list.

[0154] For example, with low group cardinality and small field widths, the data size of the aggregated intermediate value list can be several KB to tens of KB, while the data size of the complete detailed results can reach hundreds of MB or GB. The specific reduction in the amount of data returned depends on the user's group cardinality, field width, filter selectivity, and the intermediate result format used.

[0155] The result return module 208 writes the aggregated intermediate value list back to the specified result buffer in the host memory 102 through the PCIe endpoint interface module 201, and sends a processing completion notification to the database server 100.

[0156] After receiving the processing completion notification, the database executor 101 retrieves the list of aggregated intermediate values ​​from the result buffer and performs global merging of the local aggregated values ​​corresponding to different data blocks according to the user identifier on the software side. Simultaneously, the database executor 101 can continue to execute software execution fragments retained in the query plan, such as complex business rule validation, permission filtering, or output format conversion, and merge the software execution results with the global aggregated results to generate the final query response and return it to the client.

[0157] The aforementioned hardware-software co-processing flow helps shorten the data processing and data transmission time for complex statistical queries, and can free up some host CPU and bus resources for other concurrent transactions. This example illustrates that the database query hardware acceleration method described in this application can meet transaction consistency requirements while also considering data transmission efficiency and query computation performance.

[0158] It should be understood that the above order statistics scenario is merely an illustrative example. The database query hardware acceleration method described in this application can also be applied to log analysis, IoT time-series data processing, financial risk control report generation, network traffic analysis, and other business scenarios with large-scale data scanning, filtering, aggregation, connection, or sorting requirements.

[0159] The above description is merely an exemplary embodiment of this application, and the scope of protection of this application is not limited to the above embodiments. Equivalent changes, substitutions, or adjustments made by those skilled in the art based on the disclosure of this application without departing from the technical concept of this application should fall within the scope of protection of this application. The scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A hardware acceleration method for database queries, characterized in that, include: Obtain a query plan for the target data, and identify hardware-executable query segments from the query plan; Determine the data migration cost index and result compression benefit index corresponding to the query segment; Based on the data migration cost index and the result compression benefit index, determine the hardware offloading evaluation value; In response to the hardware offload evaluation value meeting the preset conditions, an acceleration descriptor containing transaction version verification information is generated and the acceleration descriptor is sent to the hardware acceleration device. Receive the intermediate compression result returned by the hardware acceleration device, and generate a query response based on the intermediate compression result; The intermediate compression result is generated by the hardware acceleration device after performing consistency verification on the acquired data blocks based on the transaction version verification information.

2. The method according to claim 1, characterized in that, The determination of the data migration cost metrics and result compression benefit metrics corresponding to the query segment includes: The data relocation cost index is determined based on the data read transmission overhead and the data return transmission overhead. Based on the estimated amount of original result data and the estimated amount of compressed result data, the result compression benefit index is determined.

3. The method according to claim 2, characterized in that, The determination of the hardware offloading evaluation value based on the data migration cost index and the result compression benefit index includes: Obtain the host processing load status and hardware pipeline load status; Based on the host processing load status and the hardware pipeline load status, determine the time saved by calculation, the time increased by relocation, and the time saved by compression. Based on the calculated time savings, the increased time from relocation, and the time saved from compression, the hardware offloading evaluation value is calculated using the following formula: in, The hardware unloading evaluation value, To save time in the calculation, Add time to the relocation. This saves time for the compression. The time saved in calculation is determined based on the host processing load status and the estimated computational load of the query segment; the time increase in relocation is determined based on the data read transmission overhead, the data return transmission overhead, and the hardware pipeline load status; the time saved in compression is determined based on the estimated raw result data volume, the estimated compressed result data volume, and the bus bandwidth.

4. The method according to claim 1, characterized in that, The acceleration descriptor also includes result return format indication information; The compressed intermediate results are intermediate state data of an incomplete result set; The receiving of the intermediate compression result returned by the hardware acceleration device includes: Receive the intermediate state data generated by the hardware acceleration device according to the result feedback format instruction information.

5. The method according to claim 4, characterized in that, The intermediate state data includes at least one of the following: When the query segment is a filtered query, the intermediate state data is a filtered bitmap; When the query segment is an aggregate query, the intermediate data is the aggregated intermediate value; When the query segment is a join query, the intermediate state data is a join matching index; When the query segment is a sorted query, the intermediate state data is a sorted index or a Top-N candidate set.

6. The method according to claim 1, characterized in that, The generation of the accelerated descriptor containing transaction version verification information includes: Dynamically generate transaction snapshot identifiers based on the current transaction view; The transaction snapshot identifier is used as the transaction version verification information and encapsulated into the acceleration descriptor; The transaction snapshot identifier includes at least one of a data page version number or a log sequence number.

7. The method according to claim 1, characterized in that, The step of identifying hardware-executable query segments from the query plan includes: The query fragment containing at least one of the following operators is identified from the query plan as the hardware-executable query fragment: data block scan operator, predicate filtering operator, field projection operator, expression evaluation operator, hash aggregation operator, semi-join operator, hash join operator, Top-N filtering operator, and local sorting operator; The method further includes: Retain the software execution segments in the query plan, excluding the hardware-executable query segments; The step of generating a query response based on the compressed intermediate results includes: The compressed intermediate result is merged with the execution result of the software execution fragment to generate the query response.

8. A hardware acceleration method for database queries, applied to a hardware acceleration device, characterized in that, include: The receiving host sends an acceleration descriptor, which includes transaction version verification information and result return format indication information; Data blocks are obtained based on the acceleration descriptor via the direct memory access channel; Based on the transaction version verification information, the data block is subjected to consistency verification; In response to the successful consistency check, hardware query processing is performed on the data block to generate a compressed intermediate result with an incomplete result set; The compressed intermediate result is sent back to the host according to the result return format instruction information.

9. The method according to claim 8, characterized in that, The method further includes: In response to the failure of the consistency check, a reread instruction or a software execution instruction is returned to the host; The hardware query processing of the data block includes: The data blocks are rearranged on-chip to convert them into a data stream adapted to the hardware pipeline; The data stream is processed through the hardware pipeline.

10. A hardware acceleration system for database queries, characterized in that, include: A database server configured to perform the method as described in any one of claims 1 to 7; The hardware acceleration device is connected to the database server via a peripheral interconnect bus and is configured to perform the method as described in claim 8 or 9.

Images