A database query processing method based on a network card runtime environment

Abstract

The application discloses a database query processing method based on a network card runtime environment. The method is applied to a storage node in a distributed database system, the storage node is configured with a remote direct memory access network interface card, and a remote direct memory access runtime environment is run on the network interface card. The method comprises the following steps: receiving intermediate representation code and query parameter values sent by a computing node, wherein the intermediate representation code is generated by the computing node according to the computing logic in a database query plan; converting the intermediate representation code into computing instructions executable by the network interface card through the runtime environment and executing the computing instructions to filter, aggregate and / or column-select the memory data of the storage node according to the query parameter values, thereby generating a computing result; and writing the computing result into a result buffer and sending an execution completion notification to the computing node, so that the computing node reads the computing result. The method avoids the participation of the central processing unit of the storage node host in the query computing process, thereby reducing the computing load of the central processing unit.

Description

Technical Field

[0001] This invention relates to the field of distributed database technology, and in particular to a database query processing method based on the network interface card (NIC) runtime environment. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, distributed database systems are being used more and more widely. To cope with the demands of massive data storage and high-concurrency access, distributed databases generally adopt a compute-storage separation architecture. Compute nodes are responsible for parsing, optimizing, and generating query plans using the Structured Query Language (SQL), while storage nodes are responsible for data storage and basic data scanning. The two communicate via a high-speed network. This architecture enables independent elastic scaling of compute and storage resources, reducing system maintenance costs and hardware requirements. However, the separation of compute and storage also brings new performance bottlenecks—during query execution, storage nodes need to transmit large amounts of raw data to compute nodes over the network, where the compute nodes perform filtering, aggregation, projection, and other computational operations. As data volumes continue to grow, network transmission overhead has become one of the key factors restricting the query performance of distributed databases.

[0003] To alleviate network transmission pressure, existing technologies commonly employ predicate pushdown, where filtering conditions (such as comparison and logical operations) generated by the compute nodes are distributed to the storage nodes. The storage node's CPU then performs the filtering during data scanning, returning only data that meets the conditions to the compute nodes over the network. However, in scenarios with limited selectivity in the filtering conditions or extremely large datasets, predicate pushdown can only reduce the amount of data, and network bandwidth bottlenecks and high latency issues remain prominent. Furthermore, the storage node's CPU must handle the execution of the pushed-down filtering logic; in high-concurrency query scenarios, the CPU may become another bottleneck restricting overall system performance.

[0004] Remote Direct Memory Access (RDMA) technology can bypass the operating system kernel and host CPU to achieve low-latency, high-bandwidth data exchange between storage nodes and compute nodes, and has been applied in the field of distributed databases. Existing solutions primarily use RDMA as an efficient data transmission channel to accelerate data transfer between nodes or memory data read / write operations. Although the atomic operations of the RDMA protocol have been proven to be Turing complete, theoretically enabling general-purpose computations on network interface cards, related technologies have not yet effectively utilized this capability to offload computational tasks in database queries. Core computational logic such as database filtering and aggregation still relies on the host CPU. Due to this limitation, existing distributed database systems still have significant room for optimization in terms of query latency, network throughput, and CPU load. Summary of the Invention

[0005] This invention provides a database query processing method based on the network interface card runtime environment to solve the problems of high network transmission overhead, high query latency, and heavy load on the central processing unit of the storage node host caused by the need to transmit a large amount of raw data to the computing node for processing in the existing distributed database system with a computing and storage separation architecture.

[0006] According to one aspect of the present invention, a database query processing method based on a network interface card (NIC) runtime environment is provided, applied to a storage node in a distributed database system, wherein the storage node is configured with a remote direct memory access (RDA) network interface card, and the RDA runtime environment runs on the network interface card; the method includes: The system receives intermediate representation code and query parameter values ​​sent by the computing node. The intermediate representation code is generated by the computing node based on the computation logic in the database query plan. The intermediate representation code includes parameterized filtering conditions and / or aggregation logic. The intermediate representation code is converted into executable computation instructions by the network interface card through the runtime environment and executed to filter, aggregate, and / or perform column selection calculations on the memory data of the storage node according to the query parameter values, and generate computation results. The calculation result is written to the result buffer, and an execution completion notification is sent to the computing node so that the computing node can read the calculation result.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the database query processing method based on the network interface card runtime environment as described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the database query processing method based on the network interface card runtime environment as described in any embodiment of the present invention.

[0009] This invention utilizes a runtime environment running on a Remote Direct Memory Access Network Interface Card (RDANIC) on a storage node to convert intermediate representation code issued by the compute node into executable computation instructions for the DCANIC and execute them on the DCANIC. Database query computation tasks such as filtering and aggregation are offloaded from the central processing unit (CPU) of the host computer where the storage node resides to the DCANIC of the storage node for execution. Only the computation results, rather than the original data, are returned to the compute node. This reduces the amount of network data transmission between nodes, lowers query response latency, and avoids the CPU of the storage node host computer participating in the query computation process, reducing the computational load on the host CPU and improving the overall throughput of the distributed database system in high-concurrency query scenarios.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a database query processing method based on the network card runtime environment provided in an embodiment of the present invention; Figure 2 This is a flowchart of a database query processing method based on the network card runtime environment provided in an embodiment of the present invention; Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] For ease of description, in the specification, claims, and drawings of this invention, "network interface card" refers to a remote direct memory access network interface card, and "runtime environment" refers to a remote direct memory access runtime environment running on the network interface card. The network interface card is configured on the storage node of a distributed database system and possesses remote direct memory access communication capabilities and Turing-complete atomic operation execution capabilities. The runtime environment refers to the remote direct memory access runtime environment running on the network interface card.

[0016] Figure 1 This is a flowchart of a database query processing method based on a network interface card (NIC) runtime environment provided by an embodiment of the present invention. This embodiment is applicable to distributed database systems with a compute-storage separation architecture, where the query pushdown computation task is performed by the Remote Direct Memory Access Network Interface Card (RNIC) of the storage node. The method is applied to the storage node in the distributed database system, wherein the storage node is configured with a Remote Direct Memory Access Network Interface Card (RNIC), and the NIC runs a Remote Direct Memory Access Runtime Environment (RRE). Figure 1 As shown, the method includes: S110. Receive intermediate representation code and query parameter values ​​sent by the computing node. The intermediate representation code is compiled and generated by the computing node according to the computing logic in the database query plan. The intermediate representation code includes parameterized filtering conditions and / or aggregation logic.

[0017] A Remote Direct Memory Access Runtime Environment (RRE) is a software runtime system running on a Remote Direct Memory Access Network Interface Card (RNIC) used to receive, translate, and execute database query computation logic. This runtime environment includes functional modules such as a just-in-time (JIT) compiler and / or interpreter, a context manager, a secure memory access unit, and a result output unit. It can convert the intermediate representation code issued by the compute node into a sequence of instructions directly executable by the RNIC and execute it, or directly interpret and execute the intermediate representation code.

[0018] Intermediate representation code refers to intermediate-form program code, independent of specific hardware instruction sets, compiled by compute nodes based on the computational logic in the database query plan. Intermediate representation code includes parameterized filtering conditions and / or aggregation logic, and is platform-independent and portable. Query parameter values ​​refer to the specific runtime binding values ​​of placeholders (such as "?") in Structured Query Language (SQL) statements, obtained by the compute node during the compilation of the intermediate representation code and distributed along with it.

[0019] When a client submits an SQL query to the distributed database system, the compute nodes perform lexical, syntactic, and semantic analysis on the SQL statement to generate a logical execution plan. The compiler within the compute nodes extracts the computational logic involving data filtering and / or aggregation from this execution plan and compiles it into intermediate representation code. The filtering conditions and / or aggregation logic in this intermediate representation code are expressed in parameterized form; that is, comparison values ​​in the conditions, boundary values ​​for aggregation, etc., exist as placeholders or variable references, rather than hard-coded specific values. The compute nodes simultaneously obtain the specific binding values ​​of the placeholders at runtime as query parameter values ​​and send the intermediate representation code and query parameter values ​​together to the storage nodes.

[0020] S120. The intermediate representation code is converted into executable calculation instructions for the network interface card through the runtime environment and executed to filter, aggregate and / or select columns for the memory data of the storage node according to the query parameter value, and generate calculation results.

[0021] After receiving the intermediate representation code, the remote direct memory access (RDA) runtime environment of the storage node converts it into computation instructions that the RDA network interface card can directly execute. The RDA network interface card then executes these computation instructions. During execution, the runtime environment performs various calculations on the data stored in the storage node's memory based on the specific values ​​of the query parameters issued by the compute node. These calculations include filtering (selecting data rows that meet the requirements), aggregation (performing statistical operations such as counting and summing on data rows that meet the requirements), and / or column selection (reading only the columns required for the query and ignoring unnecessary columns). The resulting calculations generate the corresponding results.

[0022] Optionally, the step of converting the intermediate representation code into computational instructions executable by the network interface card through the runtime environment includes: compiling the intermediate representation code into a sequence of instructions directly executable by the network interface card at runtime using a just-in-time compiler built into the runtime environment, and performing constant folding optimization based on the query parameter value; caching the instruction sequence in the local cache of the network interface card or in the reserved memory of the storage node so that the instruction sequence can be reused when receiving intermediate representation code with the same logical structure but different query parameter values.

[0023] A Just-In-Time (JIT) compiler is a built-in compilation component of the runtime environment. Its function is to convert platform-independent intermediate representation code into a sequence of machine instructions that can be directly executed by the network interface card at runtime. Unlike traditional static compilation (which completes compilation before query execution), JIT compilation occurs at the query execution time. The compilation process can utilize the specific values ​​of the query parameters known at runtime for targeted optimization. One key optimization is constant folding optimization: during compilation, the JIT compiler treats the received query parameter values ​​as constants that can be determined at compile time. It pre-computes expressions or operands in the intermediate representation code whose results can be determined during compilation, and directly embeds the calculation results into the generated instruction sequence, replacing the instruction sequence that would otherwise require reading the parameter values ​​and performing calculations at runtime. For example, when the filter condition is "age>?" and the query parameter value is "25", the JIT compiler directly embeds the parameter value "25" as an immediate value into the operand of the comparison instruction. At runtime, the comparison operation can be performed directly without reading the parameter value from registers or memory, thereby reducing the number of runtime instructions and memory accesses.

[0024] Optionally, the compiled instruction sequence can be cached in the local cache of the network interface card or in the reserved memory of the storage node, so that the instruction sequence can be reused when intermediate representation code with the same logical structure but different query parameter values ​​is received. The just-in-time compiler caches the instruction sequence generated each time it is compiled. The cache location can be the onboard local cache of the network interface card (such as the cache memory inside the network card) or a dedicated memory area reserved for the runtime environment in the host memory of the storage node. When a new intermediate representation code issued by the compute node in a subsequent query has the same logical structure as the intermediate representation code corresponding to the cached instruction sequence (i.e., the calculation logic of filtering conditions, aggregation types, and data columns involved is exactly the same), and only the query parameter values ​​are different, the runtime environment does not need to re-execute the compilation process. It can directly load the compiled instruction sequence from the cache and only substitute the new query parameter values ​​as input data for execution. Through the above cache reuse mechanism, for multiple query requests with only different query parameter values ​​in the same query mode, just-in-time compilation only needs to be performed once during the first query. Subsequent queries directly reuse the compilation results, which significantly reduces the computational overhead and compilation time caused by repeated compilation, further reduces query latency, and improves the overall throughput of the system.

[0025] Optionally, before converting the intermediate representation code into executable computation instructions for the network interface card through the runtime environment and executing it, the method further includes: performing security verification on the received intermediate representation code, the security verification including: verifying whether the intermediate representation code only includes allowed opcodes, detecting whether the loop structure in the intermediate representation code exceeds a preset maximum iteration limit; verifying whether the computing node has the permission to issue programs and access the result buffer according to the remote direct memory access permission control mechanism; the permission control mechanism includes protection domain verification and memory key verification; after the verification is passed, allocating an independent memory sandbox area in the memory of the storage node for the execution process of the intermediate representation code, the execution process can only read and write data in the memory sandbox area, and is prohibited from accessing storage node memory data outside the memory sandbox area.

[0026] Optionally, since the intermediate representation code is compiled by the compute nodes and distributed to the storage nodes, its content may contain malicious or illegal operation logic due to configuration errors or attacks on the compute nodes. Therefore, the intermediate representation code needs to be security verified before it is executed in the runtime environment to ensure the security and stability of the execution process.

[0027] Security verification includes: verifying whether the intermediate representation code contains only allowed opcodes, and detecting whether the loop structure in the intermediate representation code exceeds a preset maximum iteration count. Optionally, the runtime environment has a preset set of allowed opcodes, which limits the types of legal instructions that the runtime environment can recognize and execute. The runtime environment checks each opcode in the received intermediate representation code. If any opcode is found to be outside the set of allowed opcodes, the intermediate representation code is deemed illegal, execution is refused, and an error message is returned to the computing node. Simultaneously, the runtime environment performs static analysis on the loop structure in the intermediate representation code to extract the maximum number of iterations for the loop. If this number exceeds a preset maximum iteration count, execution is refused, and an error message is returned to the computing node.

[0028] Furthermore, based on the remote direct memory access (RDA) permission control mechanism, the system verifies whether the compute node has the authority to send programs and access the result buffer. The permission control mechanism includes protection domain verification and memory key verification. During the preparation phase before query execution, the compute node and storage node establish a RDA connection and negotiate to determine the memory regions used for sending programs and receiving results. Only compute nodes that pass both protection domain verification and memory key verification are allowed to send intermediate representation code and access data in the result buffer to the storage node. If verification fails, execution is refused, and a permission error message is returned to the compute node.

[0029] After both security and permission verifications are passed, a separate memory sandbox area is allocated in the memory of the storage node for the execution process of the intermediate representation code. The execution process of the intermediate representation code can only read and write data within the memory sandbox area; access to storage node memory data outside the memory sandbox area is prohibited. Through layers of protection including opcode verification, loop limit detection, protection domain verification, memory key verification, and memory sandbox isolation, system security and data security are ensured when the storage node executes dynamic computation logic issued by the compute node.

[0030] S130. Write the calculation result into the result buffer and send an execution completion notification to the computing node so that the computing node can read the calculation result.

[0031] After completing all computation operations, the remote direct memory access (RDA) runtime environment of the storage node writes the generated computation results into a pre-negotiated result buffer in the storage node's memory. The location and size of the result buffer are negotiated and determined by the compute node and storage node during the preparation phase before query execution, and the RDA key and protection field mechanism ensures that only legitimate compute nodes can access the result buffer. The runtime environment sends an execution completion notification to the compute node through the RDA completion queue mechanism or the RDA send operation. Upon receiving the notification, the compute node reads the computation results from the result buffer through the RDA read operation and performs subsequent processing (such as returning the query results to the client).

[0032] This invention utilizes a runtime environment running on a Remote Direct Memory Access Network Interface Card (RDANIC) on a storage node to convert intermediate representation code issued by the compute node into executable computation instructions for the DCANIC and execute them on the DCANIC. Database query computation tasks such as filtering and aggregation are offloaded from the central processing unit (CPU) of the host computer where the storage node resides to the DCANIC of the storage node for execution. Only the computation results, rather than the original data, are returned to the compute node. This reduces the amount of network data transmission between nodes, lowers query response latency, and avoids the CPU of the storage node host computer participating in the query computation process, reducing the computational load on the host CPU and improving the overall throughput of the distributed database system in high-concurrency query scenarios.

[0033] Figure 2 This is a flowchart of a database query processing method based on a network interface card (NIC) runtime environment provided by an embodiment of the present invention. This embodiment optimizes and improves the process of "converting the intermediate representation code into executable calculation instructions for the network interface card through the runtime environment and executing them" based on the above embodiments. For example... Figure 2 As shown, the method includes: S210. Receive intermediate representation code and query parameter values ​​sent by the computing node. The intermediate representation code is compiled and generated by the computing node according to the computing logic in the database query plan. The intermediate representation code includes parameterized filtering conditions and / or aggregation logic.

[0034] S220. Using the interpreter built into the runtime environment, read the instructions in the intermediate representation code one by one, and call the corresponding remote direct memory access primitive mapping function according to the operation type of each instruction, so as to filter, aggregate and / or select columns to calculate the memory data of the storage node according to the query parameter value, and generate the calculation result.

[0035] The operation types include: memory read operation, compare and swap atomic operation, read and increment atomic operation, result write operation, conditional jump operation, and loop jump operation.

[0036] Optionally, the interpreter maintains virtual registers, a program counter, and temporary variables during execution, executes instructions sequentially according to the pointer of the program counter, and modifies the value of the program counter to achieve control flow jump when encountering the conditional jump operation or the loop jump operation; the remote direct memory access primitive mapping function includes: a first mapping function, used to read data from the memory of the storage node at a specified address in response to the memory read operation; a second mapping function, used to compare the value at the specified memory address with the expected value in response to the compare and swap atomic operation, and replace the value at the memory address with the target value if they are equal; a third mapping function, used to read the value at the specified memory address and write it back after incrementing the value by a specified step in response to the read and increment atomic operation; and a fourth mapping function, used to write the calculation result to the result buffer in response to the result write operation.

[0037] The intermediate representation code can be converted into computational instructions executable by the network interface card through the runtime environment and then executed. Alternatively, it can be executed through an interpreter, as follows: The runtime environment's built-in interpreter reads instructions from the intermediate representation code line by line. Based on the operation type of each instruction, it calls the corresponding remote direct memory access primitive mapping function to filter, aggregate, and / or perform column selection calculations on the memory data of the storage nodes according to the query parameter values, generating the calculation results. The operation types include: memory read operations, compare-and-swap atomic operations, read-and-add atomic operations, result write operations, conditional jump operations, and loop jump operations.

[0038] Optionally, the interpreter maintains virtual registers, a program counter, and temporary variables during execution. Instructions are executed sequentially according to the program counter's pointer, and the program counter's value is modified to implement control flow jumps when encountering conditional jumps or loop jumps. The remote direct memory access primitive mapping functions include a first mapping function, a second mapping function, a third mapping function, and a fourth mapping function. The first mapping function reads data from a specified address in the memory of a storage node in response to a memory read operation. The second mapping function compares the value at a specified memory address with the expected value in response to a compare-and-swap atomic operation; if they are equal, the value at that memory address is replaced with the target value. The third mapping function reads the value at a specified memory address, increments it by a specified step, and writes it back in response to a read-and-increment atomic operation. The fourth mapping function writes the calculation result to the result buffer in response to a result write operation.

[0039] For example, when conditional judgments (such as comparison operations like greater than, less than, and equal to in a WHERE clause) are required, the interpreter implements conditional branching by calling the second mapping function (comparing and swapping atomic operations). When it is necessary to traverse multiple rows of records in the data table, the interpreter implements loop control by modifying the program counter and repeatedly reads the next row of data in conjunction with the first mapping function. When it is necessary to count or sum records that meet the conditions, the interpreter implements cumulative statistics by calling the third mapping function (reading and incrementing atomic operations). The interpreter converts the database calculation logic in the intermediate representation code line by line into a sequence of calls to remote direct memory access atomic operations and memory access operations, and performs data filtering, aggregation, and / or column selection calculations directly on the network interface card of the storage node, without generating independent instruction sequences or requiring the central processing unit of the storage node to participate in the execution.

[0040] By reading and executing intermediate representation code line by line by line through the interpreter, the startup latency of query execution is reduced without the need for pre-compiling and generating independent instruction sequences. At the same time, the execution context is maintained through virtual registers, program counters and temporary variables. In conjunction with remote direct memory access primitive mapping functions, database operations are directly converted into atomic operations and memory access operations of the network interface card. This enables computational tasks such as filtering and aggregation to be completed efficiently on the network interface card of the storage node, further reducing the burden on the central processing unit of the storage node host.

[0041] In some embodiments, the intermediate representation code employs an instruction set based on remote direct memory access (RDM) primitives. This instruction set includes the following types of instructions: register load instructions, used to load data acquired by a RDM read operation into a virtual register; register store instructions, used to store data from a virtual register into a specified memory address; comparison instructions, used to compare the values ​​in two virtual registers and set a flag; branch jump instructions, used to determine whether to jump to a target address based on a flag; and loop control instructions, used to implement loop traversal operations. When executing the above instructions, the interpreter calls the corresponding RDM primitive mapping function according to the operation type of the instruction. For example, the first mapping function is called for a memory read operation, the second mapping function is called for a compare-and-swap atomic operation, the third mapping function is called for a read-and-increment atomic operation, and the fourth mapping function is called for a result write operation. Through the correspondence between the above instruction set and mapping functions, the database calculation logic in the intermediate representation code is converted line by line into a sequence of RDM operations executable by the network interface card.

[0042] S230. Write the calculation result into the result buffer and send an execution completion notification to the computing node so that the computing node can read the calculation result.

[0043] Optionally, the step of filtering, aggregating, and / or selecting columns in the memory data of the storage node according to the query parameter value to generate a calculation result includes: according to the control flow of the calculation instruction, sequentially traversing the data rows to be queried in the memory of the storage node, performing a filtering predicate judgment on each data row; writing the identifier or position information of the data row that meets the filtering conditions into the result buffer, and / or, performing a counting or summation operation on the data rows that meet the filtering conditions and then writing the aggregation result into the result buffer; wherein, the calculation instruction implements the comparison operation, logical operation, and arithmetic operation in the filtering conditions by combining the comparison and swap atomic operation and the read and increment atomic operation of remote direct memory access.

[0044] Optionally, according to the control flow of the calculation instructions, the rows of data to be queried in the memory of the storage node are traversed sequentially, and a filtering predicate judgment is performed on each row of data; the identifier or position information of the data row that meets the filtering conditions is written into the result buffer, and / or, after performing a counting or summation operation on the data rows that meet the filtering conditions, the aggregation result is written into the result buffer. The calculation instructions implement the comparison, logical, and arithmetic operations in the filtering conditions by combining and calling the compare-and-swap atomic operation and the read-and-increment atomic operation of remote direct memory access.

[0045] The control flow includes sequential execution, conditional branching, and loop control. For each row of data to be queried, the calculation instruction extracts the value of a specified field from that row, compares this value with the threshold in the filtering conditions using a comparison and swap atomic operation, and determines whether the row meets the filtering conditions. For complex filtering conditions involving logical operations, multiple comparison and swap atomic operations are executed sequentially, and the results of each operation are logically merged. For data rows that meet the filtering conditions, the corresponding data row identifier or position information is written to the result buffer, and / or, a count accumulation or summation is performed using a read and increment atomic operation, and the aggregated result is written to the result buffer. Comparison operations, logical operations, arithmetic operations, and result writing are all completed by combining comparison and swap atomic operations and read and increment atomic operations, without the need for the host CPU of the storage node to participate.

[0046] By performing database query filtering, aggregation, and column selection calculations on the remote direct memory access network interface card (RDBMI) of the storage node, the query calculation task is completed directly at the location of the data. Only the result data that meets the conditions is returned to the compute node, eliminating the need to transmit the original data over the network to the compute node for processing. This significantly reduces the amount of network data transmission between the compute node and the storage node, and lowers query response latency. At the same time, the calculation process is completed on the network interface card of the storage node, and the execution process does not depend on the host CPU of the storage node. This frees up the host CPU of the storage node from query calculation tasks, allowing it to handle more concurrent query requests and other storage management tasks, thereby improving the overall throughput and resource utilization of the distributed database system in high-concurrency query scenarios.

[0047] Optionally, the calculation result includes only data that satisfies the filtering conditions and / or aggregation logic; the calculation result may be represented in the form of data row identifiers, bitmaps, and / or aggregation results.

[0048] The calculation results only include data that meets the filtering conditions and / or aggregation logic, excluding unfiltered raw memory data. Data row identifiers are unique identifiers (such as row numbers or primary key values) for data rows that meet the filtering conditions within the data table. After receiving the list of data row identifiers, the compute node can further retrieve complete data rows from the storage node as needed. The bitmap is a Boolean mapping of the positions of data rows that meet the filtering conditions. Each bit in the bitmap corresponds to a row in the data table; a first value (e.g., "1") indicates that the corresponding row meets the filtering conditions, and a second value (e.g., "0") indicates that the corresponding row does not meet the filtering conditions. After receiving the bitmap, the compute node can directly determine which rows meet the conditions without parsing the specific data. The aggregation result is the numerical result obtained after performing statistical operations such as counting or summing on the data rows that meet the filtering conditions. Data row identifiers, bitmaps, and aggregation results can be used individually or in combination, for example, returning both the list of data row identifiers and the aggregation result simultaneously. By returning only the calculation results instead of the raw data, the amount of network data transmission between the compute node and the storage node is further reduced, lowering query latency and bandwidth consumption.

[0049] This invention, through its embodiments, reads intermediate representation code line by line by line through an interpreter and directly executes it by calling the corresponding remote direct memory access primitive mapping function. This eliminates the need for pre-compilation into a sequence of machine instructions, reducing compilation waiting time before query initiation and lowering the execution overhead of a single query. Simultaneously, it maintains a complete execution context through virtual registers, a program counter, and temporary variables. Combined with atomic operations such as compare-and-swap and read-and-increment, it directly manipulates the memory data of the storage node. This allows computational tasks such as filtering and aggregation to be completed instantly on the network interface card of the storage node. This avoids dependence on the host CPU and can flexibly adapt to query scenarios with frequent changes in computational logic, dynamic binding of parameter values, and variable query structures.

[0050] Optionally, after sending an execution completion notification to the computing node, the method further includes: after the computing node obtains the computation result from the result buffer through a remote direct memory access read operation, releasing the memory resources occupied by the result buffer, clearing the instruction sequence cached by the just-in-time compiler or the intermediate state data cached by the interpreter, and reclaiming the memory sandbox area; and when an exception occurs or an execution timeout occurs during the execution process, returning an error message to the computing node and suspending the current execution.

[0051] In some embodiments, after sending an execution completion notification to the compute node, the process further includes resource reclamation and exception handling steps. Specifically, after the compute node retrieves the computation result from the result buffer via a remote direct memory access read operation, the runtime environment releases the memory resources occupied by the result buffer, clears the instruction sequence cached by the just-in-time compiler or the intermediate state data cached by the interpreter, and reclaims the memory sandbox area for reuse by subsequent query tasks. This resource reclamation mechanism avoids the cumulative occupation of storage node memory resources after multiple query executions, ensuring the long-term stable operation of the system.

[0052] Meanwhile, if an exception occurs during execution (such as memory access exceeding bounds, execution of illegal opcodes, failure of atomic operations, etc.) or the execution time exceeds the preset timeout threshold, the runtime environment immediately suspends the current execution, returns an error message to the compute node, and cleans up the temporary resources occupied during the current execution. This exception handling mechanism prevents erroneous or malicious code from occupying storage node resources for extended periods, enhancing the system's robustness and security.

[0053] In some embodiments, when multiple query tasks are executed concurrently, the runtime environment maintains an independent execution context for each query task. The execution context includes: an instruction sequence cache generated by the just-in-time compiler, a set of virtual registers, a program counter, a pointer to the currently processed data row, and query parameter values. Each execution context is isolated from the others and does not share data; the execution state of one query task does not affect the execution state of other query tasks. Through this independent context management mechanism, the runtime environment can process multiple query tasks simultaneously on the network interface card, and each task maintains its own independent program counter and register state during execution, ensuring the correctness and stability of concurrent queries. Once a query task is completed, its corresponding execution context is released, and related resources are reclaimed for use by subsequent tasks.

[0054] In some embodiments, for filtering conditions containing complex expressions, the runtime environment employs a hybrid execution strategy: a simplified filtering logic is executed via the network interface card (NIC) to perform preliminary screening of the memory data on the storage node, returning the preliminary candidate data row identifiers or candidate datasets to the host CPU of the storage node; the host CPU then performs precise calculations of the complex expressions on the candidate datasets to obtain the final result. This hybrid strategy reduces the amount of data transmitted to the host CPU while ensuring computational correctness, and avoids the resource overhead of implementing overly complex computational logic on the NIC. It is suitable for scenarios where filtering conditions are extremely complex and difficult to implement entirely on the NIC.

[0055] In some embodiments, after the compute node sends the intermediate representation code and query parameter values ​​to a designated area of ​​the storage node's memory via a remote direct memory access (RDM) write operation, it also sends a RDM send message to the storage node to notify the runtime environment to begin processing the new query task. Upon receiving this send message, the runtime environment reads the intermediate representation code and query parameter values ​​from the designated area, starts a just-in-time (JIT) compiler or interpreter to execute the intermediate representation code, and writes the computation result to the result buffer after execution. Simultaneously, it notifies the compute node that the result is ready via a completion queue. This coordinated process of sending, triggering, executing, and notifying ensures execution timing synchronization between the compute node and the storage node.

[0056] According to embodiments of the present invention, the present invention also provides a computer-readable storage medium storing computer instructions for causing a computer to execute the database query processing method based on the network interface card runtime environment described in the above method embodiments.

[0057] According to embodiments of the present invention, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the database query processing method based on the network interface card runtime environment described in the above method embodiments.

[0058] The computer-readable storage medium and computer program product described above are based on the same inventive concept and have the same beneficial effects as the aforementioned method embodiments. For specific technical details, please refer to the method embodiments, which will not be repeated here.

[0059] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0060] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A database query processing method based on the network interface card (NIC) runtime environment, characterized in that, A storage node applied in a distributed database system, the storage node being configured with a remote direct memory access network interface card, the network interface card running a remote direct memory access runtime environment, the method comprising: The system receives intermediate representation code and query parameter values ​​sent by the computing node. The intermediate representation code is generated by the computing node based on the computation logic in the database query plan. The intermediate representation code includes parameterized filtering conditions and / or aggregation logic. The intermediate representation code is converted into executable computation instructions by the network interface card through the runtime environment and executed to filter, aggregate, and / or perform column selection calculations on the memory data of the storage node according to the query parameter values, and generate computation results. The calculation result is written to the result buffer, and an execution completion notification is sent to the computing node so that the computing node can read the calculation result.

2. The method of claim 1, wherein, The step of converting the intermediate representation code into computational instructions executable by the network interface card through the runtime environment includes: The intermediate representation code is compiled into an instruction sequence that can be directly executed by the network interface card at runtime using the just-in-time compiler built into the runtime environment, and constant folding optimization is performed based on the query parameter value. The instruction sequence is cached in the local cache of the network interface card or in the reserved memory of the storage node.

3. The method according to claim 1, characterized in that, The step of converting the intermediate representation code into executable computation instructions for the network interface card through the runtime environment and executing them includes: The runtime environment's built-in interpreter reads each instruction in the intermediate representation code and calls the corresponding remote direct memory access primitive mapping function according to the operation type of each instruction. The operation types include: memory read operation, compare and swap atomic operation, read and add atomic operation, result write operation, conditional jump operation, and loop jump operation.

4. The method according to claim 3, characterized in that, The method further includes: maintaining virtual registers, program counters, and temporary variables during execution through the interpreter, executing instructions sequentially according to the pointer of the program counter, and modifying the value of the program counter to achieve control flow jump when encountering the conditional jump operation or the loop jump operation; The remote direct memory access primitive mapping functions include: The first mapping function is used to read data from the memory of the storage node at a specified address in response to the memory read operation; The second mapping function, in response to the comparison and swap atomic operation, compares the value at the specified memory address with the expected value, and if they are equal, replaces the value at the memory address with the target value. The third mapping function is used to respond to the read and increment atomic operation by reading the value at the specified memory address, incrementing the value by a specified step, and then writing it back. The fourth mapping function is used to write the calculation result to the result buffer in response to the result write operation.

5. The method according to claim 1, characterized in that, Before converting the intermediate representation code into executable computation instructions for the network interface card through the runtime environment and executing it, the method further includes: The received intermediate representation code is subjected to security verification, which includes: verifying whether the intermediate representation code only includes allowed opcodes, and detecting whether the loop structure in the intermediate representation code exceeds the preset maximum number of iterations. Based on the remote direct memory access permission control mechanism, verify whether the computing node has the permission to issue programs and access the result buffer; the permission control mechanism includes protection domain verification and memory key verification; After successful verification, an independent memory sandbox area is allocated in the memory of the storage node for the execution process of the intermediate representation code. The execution process can only read and write data within the memory sandbox area and is prohibited from accessing memory data of the storage node outside the memory sandbox area.

6. The method according to claim 1, characterized in that, The step of filtering, aggregating, and / or selecting columns to calculate the memory data of the storage node based on the query parameter value, and generating calculation results, includes: According to the control flow of the calculation instruction, the data rows to be queried in the memory of the storage node are traversed sequentially, and a filtering predicate judgment is performed on each data row; the identifier or position information of the data row that meets the filtering conditions is written into the result buffer, and / or, after performing a counting or summation operation on the data rows that meet the filtering conditions, the aggregation result is written into the result buffer. The calculation instructions combine remote direct memory access comparison and swap atomic operations and read and increment atomic operations to implement the comparison, logical and arithmetic operations in the filtering conditions.

7. The method according to claim 1, characterized in that, The calculation results include only data that meets the filtering conditions and / or aggregation logic; the calculation results are presented in the form of data row identifiers, bitmaps, and / or aggregation results.

8. The method according to any one of claims 2-5, characterized in that, After sending the execution completion notification to the computing node, the process also includes: After the computing node obtains the computation result from the result buffer through a remote direct memory access read operation, it releases the memory resources occupied by the result buffer, clears the instruction sequence cached by the just-in-time compiler or the intermediate state data cached by the interpreter, and reclaims the memory sandbox area. If an exception occurs or an execution timeout occurs during the execution process, an error message is returned to the computing node, and the current execution is terminated.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the database query processing method based on the network interface card runtime environment as described in any one of claims 1-8.

10. A computer program product comprising a computer program that, when executed by a processor, implements the database query processing method based on a network interface card runtime environment according to any one of claims 1-8.

Images