A method and system for executing structured query statement aggregation calculation based on DPU

Abstract

The present invention provides a method and system for executing structured query statement aggregation calculation based on DPU, the method comprising: obtaining a structured query statement containing an aggregation function. Parsing the structured query statement, converting the structured query statement into a physical plan execution tree for DPU to load and execute according to preset rules, wherein the physical plan execution tree includes key information of aggregation calculation. Sending a cache configuration instruction to the DPU, and establishing one or more aggregation function buffers according to the key information of aggregation calculation. Scheduling the physical plan execution tree to the DPU to perform data calculation, storing the intermediate calculation results generated by the aggregation calculation during the calculation process in the corresponding aggregation function buffer for subsequent calculation calls, until the calculation result of the structured query statement is obtained. The present invention uses an aggregation function buffer to reduce the transportation of intermediate calculation results between the DPU and the host during aggregation calculation, thereby improving data transmission and data processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for executing structured query statement aggregation calculation based on DPU. Background Art

[0002] Apache Spark is a fast general-purpose computing engine designed for large-scale data processing. Spark SQL is one of Spark's computing modules specifically designed for processing structured data. Spark SQL uses the Spark framework to execute SQL queries and read and write, and all of its calculations are still based on the CPU. Therefore, in addition to maintaining the scheduling of the entire calculation, the CPU also requires additional computing power for data-intensive calculations. When Spark SQL uses the CPU to calculate high-density data on big data, computing power becomes the main bottleneck of its performance.

[0003] As a dedicated data processing chip, the DPU chip based on the KPU architecture has a very high performance improvement compared to the CPU when processing complex data calculations. Offloading SparkSQL data calculations from the CPU to the DPU can greatly improve the performance of Spark SQL and accelerate the computing power of Spark SQL in big data scenarios, allowing the CPU to focus on Spark computing scheduling and the DPU to focus on data computing in Spark SQL. In addition, the DPU computing architecture is more suitable for column computing, and some SQL statements have more advantages in column computing than row computing.

[0004] Aggregate query is a common query method in database management systems. It is used to summarize and calculate data and return a single value as a result. When executing aggregate query, various aggregate functions (such as SUM, AVG, COUNT, MAX, and MIN, etc.) are usually applied to calculate the statistical information of the data set. Aggregate query needs to save the relevant intermediate calculation results in the process of calculating the aggregate value. For example, the max function in the aggregate function needs to save the current maximum value, the count function for counting the number of data needs to save the current total number of data, the avg function for counting the average value needs to save the values ​​of count and sum at the same time, and more complex functions (such as pencentil) need to store all data. Due to the limited memory space on the DPU, when performing aggregate function calculations on the DPU based on the KPU architecture under large data volume, the data is transferred to the DPU in batches. The intermediate calculation results generated by the aggregate function in each batch during the calculation process will be transferred back to the CPU memory after the calculation is completed, and at the same time, they will be transferred back to the DPU as the input data of the next batch for calculation. Data transmission and interaction between the host memory and the memory on the DPU will consume additional resources, and when the intermediate calculation results that need to be transferred in increase, the transmission bandwidth of the new data that needs to be calculated will be occupied, resulting in reduced calculation efficiency. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and system for performing aggregate calculations of structured query statements based on DPU, so as to eliminate or improve the defect in the prior art that the intermediate calculation results generated by the aggregate calculations are transmitted back and forth between the host and the DPU, occupying transmission resources and reducing computing efficiency.

[0006] A first aspect of the present invention provides a method for performing structured query statement aggregation calculation based on a DPU, the method being executed on a host and comprising the following steps:

[0007] Get the structured query statement containing the aggregate function;

[0008] Parsing the structured query statement, and converting the structured query statement into a physical plan execution tree for DPU loading and execution according to preset rules; the physical plan execution tree includes key information of aggregate calculation;

[0009] Sending a buffer configuration instruction to the DPU to establish one or more aggregate function buffers according to the aggregate calculation key information;

[0010] The physical plan execution tree is scheduled to the DPU to perform data calculation, and the intermediate calculation results generated by the aggregation calculation during the calculation process are stored in the corresponding aggregation function buffer for subsequent calculation calls until the calculation result of the structured query statement is obtained.

[0011] In some embodiments of the present invention, the aggregate calculation key information includes: the number of the aggregate functions, the type of the aggregate function, the key column where the execution data object of the aggregate calculation is located, and the mode corresponding to the key column.

[0012] In some embodiments of the present invention, the pattern corresponding to the key column includes: column name, column data type and data length.

[0013] In some embodiments of the present invention, the aggregate function buffer is partitioned according to the key column, a corresponding aggregate function buffer is established for each key column, and the aggregate function and the pattern corresponding to the key column are stored in the header of the aggregate function buffer memory.

[0014] In some embodiments of the present invention, the aggregate function buffer is updateable, and the update method includes: a new data block enters the DPU and uses the aggregate function to calculate to obtain a new intermediate calculation result, replacing the intermediate calculation result obtained in the previous round using the same aggregate function, and updating the pattern corresponding to the key column.

[0015] In some embodiments of the present invention, the method further includes: when the used memory of the aggregate function buffer exceeds a preset value, expanding the aggregate function buffer to a preset number of times the overall memory of the aggregate function buffer.

[0016] In some embodiments of the present invention, the method further includes: returning the intermediate calculation result generated by the current calculation to the host at preset time intervals for the host to perform error detection.

[0017] The second aspect of the present invention provides a system for performing structured query statement aggregation calculations based on DPU, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer instructions, and when the computer instructions are executed, the system implements the steps of any of the methods described above.

[0018] A third aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0019] A fourth aspect of the present invention provides a computer program product, comprising a computer program, which implements the steps of any of the above methods when executed by a processor.

[0020] After the present invention opens up an aggregate function buffer on the DPU, when a structured query statement with an aggregate function is executed, the aggregate calculation will be offloaded to the DPU for calculation, and all intermediate calculation results of the aggregate function calculation will be stored in the aggregate function buffer, thereby avoiding the transfer of intermediate calculation results between the DPU and the host, avoiding repeated transmission of the same data, and improving the utilization rate of the data transmission bandwidth and the efficiency of data processing.

[0021] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0022] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0024] Figure 1 This is a flow chart of a method for performing structured query statement aggregation calculation based on DPU according to an embodiment of the present invention.

[0025] Figure 2 This is a flow chart of a method for performing structured query statement aggregation calculation based on DPU according to another embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0027] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0028] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0029] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0030] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0031] Data Processing Unit (DPU) is a hardware unit specially designed for processing data. It is usually used to accelerate data-intensive tasks such as machine learning reasoning, data analysis, image processing, etc. The functions of DPU include efficient data stream processing, parallel computing capabilities, low latency and high throughput, which help improve the efficiency and performance of data processing.

[0032] SQL (Structured Query Language) is commonly used in relational databases to represent and execute data queries. Aggregate query is a special type of database query that aims to perform aggregate functions (such as sum, average, maximum, minimum, and count) on a set of data and return a summary result. Aggregate query is usually used for statistics and analysis of data rather than simple data retrieval.

[0033] An embodiment of the present invention provides a method for executing structured query statement aggregation calculation based on DPU, the method is executed on a host, such as Figure 1 As shown, the following steps S101 to S104 are included:

[0034] Step S101: Obtain a structured query statement containing an aggregate function.

[0035] Step S102: Parse the structured query statement and convert the structured query statement into a physical plan execution tree for DPU loading and execution according to preset rules. The physical plan execution tree includes key information of aggregate calculation.

[0036] Step S103: Send a buffer configuration instruction to the DPU, and establish one or more aggregate function buffers according to the aggregate calculation key information.

[0037] Step S104: Schedule the physical plan execution tree to the DPU to perform data calculation, and store the intermediate calculation results generated by the aggregation calculation during the calculation process in the corresponding aggregation function buffer for subsequent calculation calls until the calculation result of the structured query statement is obtained.

[0038] In step S102, the physical plan execution tree is an important concept used in database systems to optimize and execute queries. It is a tree structure that describes the detailed steps and sequence of how a database system executes a specific query or operation. The execution plan tree is generated by a database query optimizer, which considers the complexity of the query, the size of the table, the index situation, and other factors to determine the most efficient way to execute the query. In the execution plan tree, each node represents an execution operation, such as table scanning, index usage, join operations, or aggregation operations. Nodes are connected by various operators, describing how data flows and is processed during execution.

[0039] In some embodiments of the present invention, the aggregate calculation key information includes: the number of aggregate functions, the type of aggregate functions, the key column where the execution data object of the aggregate calculation is located, and the mode corresponding to the key column.

[0040] The key column refers to the data column to be executed for the aggregate calculation. For example, the calculation instruction count(col1) counts the data that meets the conditions in the first column, where col1 is the key column.

[0041] The schema corresponding to the key column is an independent entity in the database. In a database management system, a schema is a description of data storage and organization, including structural information such as tables, fields, indexes, constraints, etc. It defines the logical view and organization of data objects in the database.

[0042] In some embodiments of the present invention, the schema corresponding to the key column includes: column name, column data type and data length.

[0043] In some embodiments of the present invention, the aggregate function buffer is partitioned according to key columns, a corresponding aggregate function buffer is established for each key column, and the aggregate function and the pattern corresponding to the key column are stored in the header of the aggregate function buffer memory.

[0044] In some embodiments of the present invention, the aggregate function buffer is updateable, and the update method includes: a new data block enters the DPU and is calculated using an aggregate function to obtain a new intermediate calculation result, replacing the intermediate calculation result obtained in the previous round using the same aggregate function, and updating the pattern corresponding to the key column.

[0045] In some embodiments of the present invention, the method further includes: when the used memory of the aggregate function buffer exceeds a preset value, expanding the aggregate function buffer to a preset number of times the overall memory of the aggregate function buffer.

[0046] In some embodiments of the present invention, the method further includes: returning an intermediate calculation result generated by the current calculation to the host at a preset time interval for the host to perform error detection.

[0047] For example, the host detects potential calculation errors or transmission errors by comparing the expected results with the intermediate calculation results. Another example is to compare the intermediate calculation results in different calculation cycles to ensure the consistency and correctness of the results. Or the host verifies whether the intermediate calculation results are complete, that is, whether they contain all expected data items and fields.

[0048] The present invention is described below in conjunction with another embodiment of the present invention:

[0049] When users use Spark SQL to process data, if there is an aggregate function in the SQL statement, it will be offloaded to the DPU for calculation. The existing aggregate calculation method is to read the data block into the DPU memory for calculation and processing, and transfer the temporary intermediate calculation results generated by the aggregate calculation back to the host memory. When reading the next block of data to the DPU for calculation, the temporary intermediate calculation results of the previous block of data need to be transferred back to the DPU memory.

[0050] For example, the first data block is read into the DPU memory for aggregation calculation. The aggregation function is to select the maximum value of a column. Then the maximum value of a column under this block of data will be calculated, which is the temporary intermediate calculation result. In actual situation, there may be more than one data, perhaps multiple data, or multiple aggregation functions. When the next data block read has a value larger than the previous temporary intermediate calculation result, it needs to be compared and replaced. The intermediate calculation result calculated by the new data block is returned to the host memory again and input into the DPU again as a parameter. The final result will not be obtained until all the data is processed.

[0051] It can be seen from the above process that temporary results are repeatedly moved between the host memory and the DPU memory. The data bandwidth of the DPU is limited. The more data is repeatedly moved, the lower the data processing efficiency will be, resulting in lower computing efficiency and slower speed.

[0052] After the DPU aggregate function buffer is opened on the DPU, when a SQL statement with an aggregate function is executed, the aggregate calculation will be offloaded to the DPU for calculation, and all intermediate calculation results of the aggregate function calculation will be stored in the aggregate function buffer, thereby avoiding the transfer of intermediate calculation results between the DPU memory and the host memory. Figure 2 As shown, steps S201 to S203 are included:

[0053] Step S201: When an SQL statement with an aggregate function is executed, the CPU parses the SQL statement and converts it into an executable physical plan execution tree, which contains aggregate function information. The aggregate function information specifically includes: the number of aggregate functions, the function type of the aggregate function and the key columns of the aggregate, etc.

[0054] Step S202: After obtaining the aggregate function information, through the execution of the code, the CPU will first send an instruction to the DPU to open an aggregate function buffer in the DPU's memory. The buffer will reserve a certain amount of memory space to store subsequent temporary data. The buffer will be partitioned according to the key columns. Each key column will reserve a certain amount of memory. At the same time, the aggregate function of this key column and the schema of this column will be stored at the beginning of the memory. The above together constitute the aggregate function buffer.

[0055] Step S203: After the buffer is created, when the aggregate calculation process is executed, every time a piece of data is passed in and calculated, the intermediate calculation results obtained by the DPU calculation will be stored in the DPU aggregate function buffer accordingly, and there is no need to return the temporary results of the current aggregation. When the next piece of data is passed in, there is no need to pass in the temporary calculation results of the previous piece of data again.

[0056] Furthermore, after the new data is transferred to the DPU memory and calculated, the intermediate calculation results of the current calculation will be updated to the DPU aggregation buffer. If the DPU aggregation function buffer is not enough, it will be expanded by 2 times to accommodate the scenario where the temporary results of the aggregation function are constantly growing.

[0057] The embodiment scheme is explained by taking the following table and an SQL (structured query statement) of an aggregate function as an example.

[0058] SQL statement: select id, count(name)from student group by id

[0059] Id name 1 Zhang_san 1 Li si 2 WangWu

[0060] The meaning of this SQL statement is: select the number of students for each id from the student table. The result returned is as follows:

[0061] 1 2 2 1

[0062] It means that there are 2 students with student id 1 and there is one student with student id 2.

[0063] Among them, count is an aggregation function, which means the number is obtained. Groupby is used to group the query results. In this example, it can be regarded as an aggregation condition. This statement uses ID as the aggregation condition.

[0064] The aggregate function buffer opened on the DPU uses the data in the same group as one area. In the above SQL statement, the grouping is by ID, so the data related to the ID is one buffer. If there are multiple aggregate conditions, multiple buffers will be opened.

[0065] In addition to storing the temporary intermediate calculation results of the aggregate function, the buffer also needs to store the following information:

[0066] Schema information of the aggregation buffer: For example, the data type of the key column in the zone is an integer (int), and the data type of the column name (name) is a string (String).

[0067] Data column information of the aggregation buffer: The column name of this column is (id), and the column name of the statistical column is (name)

[0068] Because the values ​​in the buffer are constantly updated as data is processed, the buffer is mutable, which facilitates the update of intermediate calculation result data. Taking the above SQL as an example, the values ​​of count and id are updated synchronously every time new data enters. When the DPU processes new data, it can directly obtain the information of all column structures from the schema and data column information in the buffer without additional input.

[0069] Corresponding to the above method, the present invention also provides a system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0070] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0071] In summary, the present invention provides a method and system for executing aggregate calculations of structured query statements based on a DPU, the method comprising: obtaining a structured query statement containing an aggregate function. Parsing the structured query statement, converting the structured query statement into a physical plan execution tree for loading and execution by the DPU according to preset rules, wherein the physical plan execution tree includes key information of aggregate calculations. Sending a cache configuration instruction to the DPU, and establishing one or more aggregate function buffers according to the key information of aggregate calculations. Scheduling the physical plan execution tree to the DPU to perform data calculations, storing the intermediate calculation results generated by the aggregate calculations during the calculation process in the corresponding aggregate function buffer for subsequent calculation calls until the calculation results of the structured query statement are obtained. The present invention uses an aggregate function buffer to reduce the transfer of intermediate calculation results between the DPU and the host during aggregate calculations, avoids repeated transmission of the same data, and improves the utilization rate of data transmission bandwidth and the efficiency of data processing.

[0072] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0073] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0074] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for performing structured query statement aggregation calculation based on DPU, characterized in that: The method is executed on a host and comprises the following steps: Get the structured query statement containing the aggregate function; Parsing the structured query statement, and converting the structured query statement into a physical plan execution tree for DPU loading and execution according to preset rules; the physical plan execution tree includes key information of aggregate calculation; Send a cache configuration instruction to the DPU, and establish one or more aggregate function buffers according to the aggregate calculation key information; the aggregate calculation key information includes: the number of the aggregate functions, the type of the aggregate function, the key column where the execution data object of the aggregate calculation is located, and the mode corresponding to the key column; the aggregate function buffer is partitioned according to the key column, a corresponding aggregate function buffer is established for each key column, and the aggregate function and the mode corresponding to the key column are stored in the head of the aggregate function buffer memory; The physical plan execution tree is scheduled to the DPU to perform data calculation, and the intermediate calculation results generated by the aggregation calculation during the calculation process are stored in the corresponding aggregation function buffer for subsequent calculation calls until the calculation result of the structured query statement is obtained.

2. The method for performing structured query statement aggregation calculation based on DPU according to claim 1, characterized in that: The mode corresponding to the key column includes: column name, column data type and data length.

3. The method for performing structured query statement aggregation calculation based on DPU according to claim 1, characterized in that: The aggregate function buffer is updateable, and the update method includes: a new data block enters the DPU and uses the aggregate function to calculate a new intermediate calculation result, which replaces the intermediate calculation result obtained in the previous round using the same aggregate function, and updates the pattern corresponding to the key column.

4. The method for performing structured query statement aggregation calculation based on DPU according to claim 1, characterized in that: The method also includes: when the used memory of the aggregate function buffer exceeds a preset value, expanding the aggregate function buffer to a preset number of times the overall memory of the aggregate function buffer.

5. The method for performing structured query statement aggregation calculation based on DPU according to claim 1, characterized in that: The method further includes: returning the intermediate calculation result generated by the current calculation to the host at preset time intervals for the host to perform error detection.

6. A system for performing structured query statement aggregation calculation based on a DPU, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instruction. When the computer program / instruction is executed, the system implements the method according to any one of claims 1 to 5. steps.

7. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images