Dpu-based data aggregation method, apparatus, device, and medium

Abstract

The present disclosure relates to a DPU-based data aggregation method, device, equipment and medium. The DPU-based data aggregation method comprises: obtaining a target query statement and target query statement corresponding to the data to be processed, starting at least one process; sending each process corresponding to the target column data to the data processor, executing the local aggregation operation corresponding to the target column data by the data processor; receiving the local aggregation result returned by the data processor, generating the result column identifier and storing it in the shared memory; further sending the result column data corresponding to the global aggregation operation to the data processor, executing the global aggregation operation by the data processor, thereby, the local aggregation operation and the global aggregation operation can be executed on the data processor, and the aggregation result is stored in the form of identifier in the shared memory, without data transmission, reducing the data transmission overhead, improving the central processing unit performance and aggregation efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to a data aggregation method, apparatus, device and medium based on DPU. Background Technology

[0002] PostgreSQL is an open-source relational database management system. After receiving an SQL query, PostgreSQL starts multiple worker processes (such as workerprocesses). When aggregation operations are involved in the execution of the SQL query, PostgreSQL's native aggregation node, the aggregate node, divides the data to be processed corresponding to the SQL query into multiple smaller chunks and distributes them to each worker process. This allows multiple worker processes to perform local aggregation operations in parallel. After each worker process completes its local aggregation operation, it passes the calculation result to the query scheduler process. The query scheduler process performs a final global aggregation on the calculation results of the local aggregation operations returned by all worker processes, thus obtaining the final aggregation result.

[0003] However, the entire aggregation operation described above is executed on the central processing unit (CPU), and there is data transfer overhead when each worker process passes the calculation results of the local aggregation operation to the query scheduling process, resulting in CPU performance degradation and low aggregation efficiency. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a data aggregation method, apparatus, device, and medium based on a DPU.

[0005] A first aspect of this disclosure provides a data aggregation method based on a DPU, comprising:

[0006] Obtain the target query statement and the corresponding data to be processed, start at least one process, and determine the target process from at least one process to execute the global aggregation operation corresponding to the target query statement;

[0007] For each process, the target column data corresponding to the process is sent to the data processor, so that the data processor can perform the local aggregation operation corresponding to the target column data, obtain the local aggregation result, and return the local aggregation result;

[0008] Receive the partial aggregation results returned by the data processor, generate the result column identifiers corresponding to the partial aggregation results, and store the result column identifiers in the shared memory of the preset database;

[0009] When it is determined that the local aggregation operations corresponding to at least one process have been completed, the target process is controlled to obtain the result column identifier corresponding to each process from the shared memory, determine the result column data corresponding to the result column identifier, and send the result column data to the data processor so that the data processor can execute the global aggregation operation and return the global aggregation result corresponding to the global aggregation operation.

[0010] Receives global aggregation results.

[0011] A second aspect of this disclosure provides a DPU-based data aggregation apparatus, comprising:

[0012] The data acquisition module is used to acquire the target query statement and the data to be processed corresponding to the target query statement, start at least one process, and determine the target process from at least one process to perform the global aggregation operation corresponding to the target query statement.

[0013] The local aggregation module is used to send the target column data corresponding to each process to the data processor, so that the data processor can perform local aggregation operations on the target column data, obtain the local aggregation result, and return the local aggregation result.

[0014] The first receiving module is used to receive the local aggregation results returned by the data processor, generate the result column identifiers corresponding to the local aggregation results, and store the result column identifiers in the shared memory of the preset database.

[0015] The global aggregation module is used to control the target process to obtain the result column identifier corresponding to each process from the shared memory when it is determined that the local aggregation operations corresponding to at least one process have been completed, and to determine the result column data corresponding to the result column identifier. The result column data is then sent to the data processor so that the data processor can execute the global aggregation operation and return the global aggregation result corresponding to the global aggregation operation.

[0016] The second receiving module is used to receive the global aggregation results.

[0017] A third aspect of this disclosure provides an electronic device, including:

[0018] processor;

[0019] Memory, used to store executable instructions;

[0020] The processor is used to read executable instructions from memory and execute the executable instructions to implement the DPU-based data aggregation method provided in the first aspect above.

[0021] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the DPU-based data aggregation method provided in the first aspect.

[0022] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0023] The data aggregation method, apparatus, device, and medium based on DPU provided in this disclosure can acquire a target query statement and the corresponding data to be processed, start at least one process, and determine a target process from the at least one process to execute the global aggregation operation corresponding to the target query statement. For each process, the target column data corresponding to the process is sent to the data processor, so that the data processor executes the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result. The local aggregation result returned by the data processor is received, and a result column identifier corresponding to the local aggregation result is generated and stored in the shared memory of a preset database. When it is determined that the local aggregation operations corresponding to at least one process have been completed, the target process is controlled to retrieve the data from the shared memory. The system retrieves the result column identifier for each process and determines the result column data corresponding to the result column identifier. This result column data is then sent to the data processor, enabling the data processor to execute a global aggregation operation and return the corresponding global aggregation result. By receiving the global aggregation result, both local and global aggregation operations can be offloaded to the data processor for execution. Furthermore, after obtaining the local aggregation result, the result column identifier corresponding to the local aggregation result is stored in the shared memory of a preset database. The target process executing the global aggregation operation can directly access the result column identifier in the shared memory, eliminating the need for other processes to transfer the local aggregation result to the target process via data transfer after completing the local aggregation operation. This reduces data transfer overhead and improves CPU performance and aggregation efficiency. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a DPU-based data aggregation method provided in an embodiment of this disclosure;

[0027] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of this disclosure;

[0028] Figure 3 This is a flowchart of another DPU-based data aggregation method provided in this embodiment of the disclosure;

[0029] Figure 4 This is a schematic diagram of the structure of a DPU-based data aggregation device provided in an embodiment of this disclosure;

[0030] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0032] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0035] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0036] Typically, the aggregation operations corresponding to existing query statements are executed on the central processing unit (CPU). Furthermore, each worker process incurs data transmission overhead when passing the calculation results of its local aggregation operations to the query scheduling process, leading to CPU performance degradation and low aggregation efficiency. To address this issue, this disclosure provides a data aggregation method based on a Data Processing Unit (DPU), which will be described below with reference to specific embodiments.

[0037] Figure 1 This is a flowchart illustrating a DPU-based data aggregation method provided in this disclosure. The method can be executed by a DPU-based data aggregation device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. Furthermore, this method can be applied to… Figure 2 The application scenario shown includes an electronic device 21 and a data processor 22. It is understood that the DPU-based data aggregation method provided in this embodiment can also be applied to other scenarios.

[0038] like Figure 2 As shown, the electronic device 21 includes a preset database, a target plugin, shared memory of the preset database, an accelerated computing engine, and worker process 1, worker process 2 and scheduling process started based on the target query statement and the preset database.

[0039] The default database can be PostgreSQL, an open-source relational database management system, or other SQL databases; there are no restrictions on this.

[0040] The target plugin can be pg-race, which is an extension of the PostgreSQL database designed to accelerate batch processing and analysis workloads using a data processor. In this embodiment, the interaction with the data processor (DPU) can be achieved based on the target plugin, thereby improving the efficiency of data aggregation processing and reducing the data processing pressure on the central processing unit.

[0041] In this embodiment of the disclosure, the target plugin can be used to call a preset interface to send column identifiers to the accelerated computing engine, wherein the column identifier can be understood as a column ID; it can also be used to store the column identifiers, the result column identifiers corresponding to the local aggregation results, and the global column identifiers corresponding to the global aggregation results in the shared memory of a preset database.

[0042] The shared memory of the default database can be understood as a memory area shared by all background processes in the default database. It is used to store various data structures that need to be shared among multiple processes. When the default database is a PostgreSQL database, the shared memory of the default database is the shared memory of the PostgreSQL database.

[0043] In this embodiment of the disclosure, the shared memory of the preset database can be used to store column identifiers, including column identifiers corresponding to the data to be processed, result column identifiers corresponding to the local aggregation results, global column identifiers corresponding to the global aggregation results, etc., for each process to call.

[0044] The accelerated computing engine can provide an interface for accessing column data. It can query and obtain the column data corresponding to the column identifier based on the column identifier sent by the target plugin. It can also call the data processor based on the preset interface so that the data processor can perform local aggregation operations and / or global aggregation operations corresponding to the target query statement. For example, the accelerated computing engine can be the Rendering And Computing Engine, abbreviated as RACE.

[0045] Worker process 1, worker process 2, and scheduler process are processes that are started after receiving the target query statement to execute the aggregation operations of the target query statement. The aggregation operations include local aggregation operations and global aggregation operations.

[0046] In this embodiment of the disclosure, the electronic device 21 and the data processor 22 can be connected by means of slot connection, network communication connection, etc.

[0047] The following is combined with Figure 2 The application scenarios shown are for Figure 1 The DPU-based data aggregation method shown will be introduced, for example, Figure 2 The electronic device 21 in the device can execute this method. For example... Figure 1 As shown, the DPU-based data aggregation method provided in this embodiment includes the following steps.

[0048] S110. Obtain the target query statement and the data to be processed corresponding to the target query statement, start at least one process, and determine the target process from the at least one process to perform the global aggregation operation corresponding to the target query statement.

[0049] In this embodiment of the disclosure, the target query statement can be a statement used to retrieve or query data from a database, wherein the specific syntax of the target query statement varies depending on the database management system used. For example, the target query statement can be a Structured Query Language (SQL) statement.

[0050] In this embodiment of the disclosure, the target query statement may be a query statement involving data aggregation, wherein data aggregation may refer to the process of merging data from multiple data sources or different data from the same data source into a single data set.

[0051] For example, the target query statement includes aggregate functions, which may include summation functions (SUM), average functions (AVG), maximum functions (MAX), minimum functions (MIN), etc.

[0052] The data to be processed can be understood as the data required to execute the target query statement. The obtained data to be processed is row-based data.

[0053] A process can be understood as the process that executes the target query statement.

[0054] The specific implementation method for starting at least one process is similar to the existing implementation method for starting a process during the execution of an SQL query statement, and will not be described in detail here.

[0055] Specifically, the electronic device can receive query requests sent by the user in real time, parse the query requests to obtain the target query statement, and then read the data to be processed corresponding to the target query statement from one or more of the following: memory, disk, preset database, etc.

[0056] S120. For each process, send the target column data corresponding to the process to the data processor so that the data processor can perform the local aggregation operation corresponding to the target column data, obtain the local aggregation result, and return the local aggregation result.

[0057] In this embodiment of the disclosure, a local aggregation operation can be understood as a partial aggregation operation, that is, a process of performing partial aggregation calculations on the data after it has been divided into blocks. This can be achieved by dividing the data into blocks based on one or more attributes.

[0058] Specifically, after obtaining the data to be processed corresponding to the target query statement, the electronic device performs block processing on the data to be processed, determines the block data corresponding to each process, and performs row-to-column conversion processing on each block data to obtain the target column data corresponding to each block data. The target column data corresponding to each process is then sent to the data processor so that the data processor can perform local aggregation operation corresponding to the target column data, obtain the local aggregation result, and return the local aggregation result. Since the data processor supports columnar data, the obtained local aggregation result is also in columnar form.

[0059] S130: Receive the local aggregation result returned by the data processor, generate the result column identifier corresponding to the local aggregation result, and store the result column identifier in the shared memory of the preset database.

[0060] In this embodiment of the disclosure, the result column identifier can be understood as the column ID corresponding to the local aggregation result, that is, the unique identification information used to characterize the local aggregation result.

[0061] Specifically, after receiving the local aggregation results returned by the data processor, the electronic device generates the result column identifiers corresponding to the local aggregation results based on a preset identifier generation method, and stores the result column identifiers in the shared memory of a preset database.

[0062] S140. When it is determined that the local aggregation operations corresponding to at least one process have been completed, the target process is controlled to obtain the result column identifier corresponding to each process from the shared memory, determine the result column data corresponding to the result column identifier, and send the result column data to the data processor so that the data processor can execute the global aggregation operation and return the global aggregation result corresponding to the global aggregation operation.

[0063] In this embodiment of the disclosure, a global aggregation operation can be understood as a process of further summarizing and calculating the local aggregation results, i.e., the intermediate result set, of the local aggregation operation after the local aggregation operation.

[0064] For example, the original aggregation function can be decomposed as follows:

[0065] min(x) = fmin(pmin(x))

[0066] max(x) = fmax(pmax(x))

[0067] sum(x) = fsum(psum(x))

[0068] count(x) = fsum(pcount(x))

[0069] avg(x)=fsum(psum()) / fsum(pcount(x))

[0070] Functions starting with 'f' represent global aggregation operations, i.e., final aggregate operations; functions starting with 'p' represent local aggregation operations, i.e., partial aggregate operations.

[0071] Specifically, after receiving the local aggregation results of the target column data corresponding to each process returned by the data processor, and / or, in response to the notification message returned by each process indicating completion, the electronic device determines that the local aggregation operations for at least one process have been completed. It then controls the target process to retrieve the result column identifier corresponding to each process from shared memory, determines the result column data corresponding to the result column identifier, and sends the result column data to the data processor. This allows the data processor to perform a global aggregation operation based on the result column data and return the global aggregation result to the electronic device. Since the data processor supports columnar data, the resulting global aggregation result is also in columnar form.

[0072] S150: Receive the global aggregation results.

[0073] In this embodiment of the disclosure, the electronic device can receive the global aggregation results returned by the data processor in real time.

[0074] In this embodiment, a target query statement and the corresponding data to be processed can be obtained. At least one process is started, and a target process for executing the global aggregation operation corresponding to the target query statement is determined from the at least one process. For each process, the target column data corresponding to the process is sent to the data processor so that the data processor executes the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result. The local aggregation result returned by the data processor is received, and a result column identifier corresponding to the local aggregation result is generated and stored in the shared memory of a preset database. When it is determined that the local aggregation operations corresponding to at least one process have been completed, the target process is controlled to retrieve the result column corresponding to each process from the shared memory. The system identifies the identifier and determines the corresponding result column data. This result column data is then sent to the data processor, enabling the data processor to execute a global aggregation operation and return the corresponding global aggregation result. The system receives the global aggregation result. This allows both local and global aggregation operations to be offloaded to the data processor for execution. Furthermore, after obtaining the local aggregation result, the result column identifier corresponding to the local aggregation result is stored in the shared memory of a preset database. The target process executing the global aggregation operation can directly access the result column identifier in the shared memory, eliminating the need for other processes to transfer the local aggregation result to the target process via data transfer after completing the local aggregation operation. This reduces data transfer overhead and improves CPU performance and aggregation efficiency.

[0075] Based on the above-described real-time example of this disclosure, the calculation process of data aggregation can be executed on the data processor, which improves the processing efficiency of data aggregation and reduces the occupation of central processing unit and memory resources. This is very advantageous for processing large amounts of data in environments with limited resources, and can also improve the processing efficiency of large amounts of data.

[0076] Based on the above embodiments of this disclosure, at least one process includes at least one worker process and one scheduler process, such as Figure 2 The worker process 1, worker process 2, and scheduler process are shown in the diagram.

[0077] In this embodiment of the disclosure, determining the target process for executing the global aggregation operation corresponding to the target query statement from at least one process may specifically include: determining whether the scheduling process has the permission to execute the global aggregation operation based on the configuration parameters corresponding to the scheduling process; if the scheduling process has the permission to execute the global aggregation operation, then the scheduling process is determined as the target process; if the scheduling process does not have the permission to execute the global aggregation operation, then the target process for executing the global aggregation operation is determined from at least one worker process based on the change value of a preset atomic variable in the shared memory.

[0078] In this embodiment of the disclosure, configuration parameters can be understood as parameters set for the scheduling process to perform aggregation operations (including global aggregation operations and / or local aggregation operations), including resource parameters, permission parameters, and other information.

[0079] Specifically, after starting the scheduling process, the electronic device obtains the configuration parameters of the scheduling process and determines whether the scheduling process has the permission to perform global aggregation operations based on the configuration parameters. If it is determined that the scheduling process has the permission to perform global aggregation operations, the scheduling process is determined as the target process; otherwise, a worker process is selected from at least one worker process and determined as the target process to perform global aggregation operations.

[0080] The specific method for selecting a worker process from at least one worker process as the target process is as follows: Atomic variables are pre-set in shared memory, where only one process (such as a worker process) can modify the value of the atomic variables at any given time. The initial value of the atomic variables is 0. When each worker process starts, the value of the atomic variables is incremented by one. Then, the worker process whose value of the atomic variables changes from 0 to 1 is determined as the target process for performing the global aggregation operation.

[0081] For example, the number of worker processes is 3. When the first worker process is started, the value of the atomic variable changes from 0 to 1. When the second worker process is started, the value of the atomic variable changes from 1 to 2. When the third worker process is started, the value of the atomic variable changes from 1 to 2. At this time, the first worker process changes the value of the atomic variable from 0 to 1. Therefore, the first worker process is determined as the target process for performing the global aggregation operation.

[0082] In this embodiment of the disclosure, when the scheduling process does not have the permission to perform global aggregation operations, a target process for performing global aggregation operations can be selected from at least one worker process, thereby improving the flexibility of process configuration and the flexibility of performing global aggregation operations.

[0083] In this embodiment of the disclosure, obtaining the data to be processed corresponding to the target query statement may specifically include: performing a data scanning operation corresponding to the target query statement, reading the data corresponding to the target query statement, and determining the data as the data to be processed; converting the data to be processed from row format to column format to obtain the column data corresponding to the data to be processed and the column identifier corresponding to each column data.

[0084] Specifically, after acquiring the target query statement, the electronic device can parse the target query statement to ensure that it conforms to a preset standard (e.g., when the database is PostgreSQL, ensure that the target query statement conforms to the SQL standard of the PostgreSQL database). After confirming that it conforms to the preset standard, it performs semantic analysis to check whether the semantics of the target query statement are correct, ensuring that the queried elements exist and conform to the semantic rules of the preset database. If it is confirmed that it conforms, it performs a query rewrite operation to optimize the query structure of the target query statement and generate multiple possible query execution plans. It selects the execution plan with the lowest cost for execution. The execution plan includes multiple nodes, each node representing a specific operation, such as scanning a table (scanning operation), joining a table, or aggregating data (aggregation operation, including at least one of local aggregation operation and global aggregation operation). After determining the execution plan with the lowest cost, it performs a data scan operation corresponding to the target query statement, reading the data corresponding to the target query statement from at least one of memory, disk, or preset database. This data is identified as the data to be processed, and the data to be processed is converted from row-based to column-based to obtain the column data corresponding to the data to be processed and the column identifier corresponding to each column.

[0085] In this embodiment of the disclosure, after obtaining the column identifier corresponding to each column of data, a mapping relationship between the column data and the column identifier is established.

[0086] In this embodiment of the disclosure, after obtaining the data to be processed, row-to-column conversion is performed on the data to be processed, so that the obtained column data can be adapted to the data format of the data processor. At the same time, column identifiers corresponding to the column data are generated and stored in shared memory, so that the column data can be directly called from the shared memory when used later, thereby reducing data transmission overhead.

[0087] Furthermore, for each process, the target column data corresponding to the process is sent to the data processor. Specifically, this may include: determining the target column identifier corresponding to the process; sending the target column identifier to the accelerated computing engine by calling the first preset interface based on the preset target plugin, so that the accelerated computing engine determines the target column data corresponding to the target column identifier based on the mapping relationship between the column identifier and the column data; and sending the target column data to the data processor by calling the second preset interface based on the accelerated computing engine.

[0088] In this embodiment of the disclosure, the target plugin can be used to store column identifiers in shared memory, and also provides an interface for calling the accelerated computing engine; the accelerated computing engine can be used to obtain column data, and can also call the data processor to enable the data processor to process aggregation operations (including global aggregation operations and local aggregation operations) on a pair of column data.

[0089] In this embodiment of the disclosure, the accelerated computing engine can send the target column data to the data processor at the same time as sending the corresponding operators to the data processor.

[0090] In this embodiment of the disclosure, column data corresponding to each process can be sent to the data processor through the target plugin and the accelerated computing engine, so that the data processor can perform data aggregation processing.

[0091] In this embodiment of the disclosure, receiving the local aggregation result returned by the data processor and generating the result column identifier corresponding to the local aggregation result, and storing the result column identifier in the shared memory of the preset database, may specifically include: receiving the local aggregation result based on the accelerated computing engine, generating the result column identifier corresponding to the local aggregation result, sending the result column identifier to the target plugin; and storing the result column identifier in the shared memory based on the target plugin.

[0092] In this embodiment of the disclosure, after generating the result column identifier corresponding to the local aggregation result, a mapping relationship between the local aggregation result and the result column identifier is established.

[0093] In this embodiment of the disclosure, the result column identifier can be stored in shared memory, which facilitates the subsequent retrieval of local aggregation results based on the result column identifier in shared memory.

[0094] Furthermore, after storing the result column identifier in the shared memory of the preset database, the DPU-based data aggregation method may also include: performing a modification operation on the flag bit corresponding to the process based on the target plugin to obtain the target flag bit, which is used to indicate that the local aggregation operation corresponding to the process has been completed.

[0095] In some embodiments of this disclosure, after storing the result column identifier corresponding to each process into shared memory, the electronic device can modify the flag bit corresponding to each process based on the target plugin, changing the first flag bit corresponding to the original execution state to the second flag bit corresponding to the execution completion state, i.e., the target flag bit, so as to notify the scheduling process that the local aggregation operation of the process has been completed.

[0096] In other embodiments of this disclosure, the electronic device can, after storing the result column identifier corresponding to each process in shared memory, control each process to send NULL data to the scheduling process, indicating that the process will no longer transmit valid data to the gather node corresponding to the scheduling process (e.g., Figure 2 In node 1), the gather node transmits the local aggregation results to the target process to perform the final global aggregation operation.

[0097] In this embodiment of the disclosure, the scheduling process can be notified of the completion of the local aggregation operation by modifying the process flag, so that the scheduling process can perform subsequent operations in response to the modification of the flag, thereby improving the effectiveness and efficiency of the aggregation operation.

[0098] In this embodiment of the disclosure, the specific implementation of determining the result column data corresponding to the result column identifier and sending the result column data to the data processor is similar to the specific implementation of sending the target column data corresponding to each process to the data processor as described above, and will not be repeated here.

[0099] In this embodiment of the disclosure, after receiving the global aggregation result, the DPU-based data aggregation method may further include: generating a global column identifier corresponding to the global aggregation result, storing the global column identifier in shared memory; converting the global aggregation result from columnar to rowar format, and controlling a scheduling process in at least one process to send the rowar data corresponding to the global aggregation result to the user, or to send the rowar data corresponding to the global aggregation result to the next node, so that the next node performs corresponding operations based on the rowar data corresponding to the global aggregation result.

[0100] Specifically, the accelerated computing engine receives the global aggregation result and generates a global column identifier corresponding to the global aggregation result. The global column identifier is sent to the target plugin, which stores the global column identifier in shared memory. At the same time, the target plugin converts the global aggregation result from columnar to rowar format. The scheduling process in at least one process is controlled to send the rowar data corresponding to the global aggregation result to the user. Alternatively, if there is a next execution node after the aggregation operation, the rowar data corresponding to the global aggregation result is transmitted to the next node so that the next node can execute the operation corresponding to the next node based on the rowar data corresponding to the global aggregation result.

[0101] In this embodiment of the disclosure, after obtaining the global aggregation result, it is possible to promptly feed back to the user or send it to the next node, so that the next node can perform subsequent operations, thereby improving the user experience and the timeliness of the next node's execution. Since the output format of the global aggregation result obtained by the DPU-based data aggregation operation is consistent with the output format of the native aggregate node of the preset database, it will not affect the native node of the preset database, thereby enhancing the compatibility and portability of the data processor in the process of executing data aggregation.

[0102] Figure 3 This is a flowchart of another DPU-based data aggregation method provided in this disclosure embodiment, such as... Figure 3 As shown, the DPU-based data aggregation method may specifically include the following steps:

[0103] S310. Obtain the target query statement, execute the data scanning operation corresponding to the target query statement, read the data corresponding to the target query statement, determine the data as the data to be processed, convert the data to be processed from row format to column format, and obtain the column format data corresponding to the data to be processed and the column identifier corresponding to each column of data.

[0104] S320. Start at least one process and determine the target process from the at least one process to perform the global aggregation operation corresponding to the target query statement.

[0105] S330. For each process, determine the target column identifier corresponding to the process, and send the target column identifier to the accelerated computing engine by calling the first preset interface based on the preset target plugin, so that the accelerated computing engine can determine the target column data corresponding to the target column identifier based on the mapping relationship between the column identifier and the column data.

[0106] S340. Based on the accelerated computing engine, the second preset interface is called to send the target column data to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result.

[0107] S350 receives local aggregation results based on the accelerated computing engine, generates result column identifiers corresponding to the local aggregation results, sends the result column identifiers to the target plugin, and stores the result column identifiers in shared memory based on the target plugin.

[0108] S360. When it is determined that the local aggregation operations corresponding to at least one process have been completed, the target process is controlled to obtain the result column identifier corresponding to each process from the shared memory, determine the result column data corresponding to the result column identifier, and send the result column data to the data processor so that the data processor can execute the global aggregation operation and return the global aggregation result corresponding to the global aggregation operation.

[0109] S370. Receive the global aggregation result, generate the global column identifier corresponding to the global aggregation result, store the global column identifier in shared memory, convert the global aggregation result from columnar to rowar format, and control the scheduling process in at least one process to send the rowar data corresponding to the global aggregation result to the user, or to send the rowar data corresponding to the global aggregation result to the next node, so that the next node can perform corresponding operations based on the rowar data corresponding to the global aggregation result.

[0110] It should be noted that the specific implementation of steps S310-S370 is similar to the implementation of the relevant steps in the above embodiments of this disclosure, and will not be repeated here.

[0111] In this embodiment, both local and global aggregation operations can be offloaded to the data processor for execution. After obtaining the local aggregation result, the result column identifier corresponding to the local aggregation result is stored in the shared memory of a preset database. The target process that performs the global aggregation operation can directly access the result column identifier in the shared memory. There is no need for other processes to transmit the local aggregation result to the target process through data transmission after the local aggregation operation is completed, which reduces data transmission overhead and improves the performance of the central processing unit and the aggregation efficiency.

[0112] Figure 4 This is a schematic diagram of the structure of a DPU-based data aggregation device provided in an embodiment of this disclosure.

[0113] In this embodiment, the DPU-based data aggregation device can be located within an electronic device and is understood as a functional module within the aforementioned electronic device. Specifically, the electronic device can be a server or a terminal, wherein the terminal specifically includes mobile phones, computers, or tablet computers, etc., without limitation.

[0114] like Figure 4 As shown, the DPU-based data aggregation device 400 may include a data acquisition module 410, a local aggregation module 420, a first receiving module 430, a global aggregation module 440, and a second receiving module 450.

[0115] The data acquisition module 410 can be used to acquire the target query statement and the data to be processed corresponding to the target query statement, start at least one process, and determine the target process from at least one process to perform the global aggregation operation corresponding to the target query statement.

[0116] The local aggregation module 420 can be used to send the target column data corresponding to each process to the data processor, so that the data processor can perform the local aggregation operation corresponding to the target column data, obtain the local aggregation result, and return the local aggregation result.

[0117] The first receiving module 430 can be used to receive the local aggregation results returned by the data processor, generate the result column identifiers corresponding to the local aggregation results, and store the result column identifiers in the shared memory of the preset database.

[0118] The global aggregation module 440 can be used to control the target process to obtain the result column identifier corresponding to each process from the shared memory when it is determined that the local aggregation operations corresponding to at least one process have been completed, and to determine the result column data corresponding to the result column identifier, and send the result column data to the data processor so that the data processor can execute the global aggregation operation and return the global aggregation result corresponding to the global aggregation operation.

[0119] The second receiving module 450 can be used to receive global aggregation results.

[0120] In this embodiment, a target query statement and the corresponding data to be processed can be obtained. At least one process is started, and a target process for executing the global aggregation operation corresponding to the target query statement is determined from the at least one process. For each process, the target column data corresponding to the process is sent to the data processor so that the data processor executes the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result. The local aggregation result returned by the data processor is received, and a result column identifier corresponding to the local aggregation result is generated and stored in the shared memory of a preset database. When it is determined that the local aggregation operations corresponding to at least one process have been completed, the target process is controlled to retrieve the result column corresponding to each process from the shared memory. The system identifies the identifier and determines the corresponding result column data. This result column data is then sent to the data processor, enabling the data processor to execute a global aggregation operation and return the corresponding global aggregation result. The system receives the global aggregation result. This allows both local and global aggregation operations to be offloaded to the data processor for execution. Furthermore, after obtaining the local aggregation result, the result column identifier corresponding to the local aggregation result is stored in the shared memory of a preset database. The target process executing the global aggregation operation can directly access the result column identifier in the shared memory, eliminating the need for other processes to transfer the local aggregation result to the target process via data transfer after completing the local aggregation operation. This reduces data transfer overhead and improves CPU performance and aggregation efficiency.

[0121] In some embodiments of this disclosure, at least one process includes at least one worker process and a scheduler process.

[0122] The data acquisition module 410 can be specifically used to determine whether the scheduling process has the permission to perform global aggregation operations based on the configuration parameters corresponding to the scheduling process; if the scheduling process has the permission to perform global aggregation operations, then the scheduling process is determined as the target process; if the scheduling process does not have the permission to perform global aggregation operations, then the target process for performing global aggregation operations is determined from at least one worker process based on the change value of the preset atomic variable in the shared memory.

[0123] In some embodiments of this disclosure, the data acquisition module 410 may also be specifically used to perform a data scanning operation corresponding to the target query statement, read the data corresponding to the target query statement, determine the data as data to be processed, convert the data to be processed from row format to column format, and obtain the column format data corresponding to the data to be processed and the column identifier corresponding to each column of data.

[0124] In some embodiments of this disclosure, the local aggregation module 420 may be specifically used to determine the target column identifier corresponding to the process, and send the target column identifier to the accelerated computing engine by calling a first preset interface based on a preset target plugin, so that the accelerated computing engine determines the target column data corresponding to the target column identifier based on the mapping relationship between the column identifier and the column data; and send the target column data to the data processor by calling a second preset interface based on the accelerated computing engine.

[0125] In some embodiments of this disclosure, the first receiving module 430 may be specifically used to receive local aggregation results based on the accelerated computing engine, generate result column identifiers corresponding to the local aggregation results, send the result column identifiers to the target plugin, and store the result column identifiers in shared memory based on the target plugin.

[0126] In some embodiments of this disclosure, the DPU-based data aggregation device 400 may further include a flag modification module.

[0127] The flag modification module can be used to modify the flags corresponding to the process based on the target plugin after storing the result column identifier in the shared memory of the preset database, so as to obtain the target flag. The target flag is used to indicate that the local aggregation operation corresponding to the process has been completed.

[0128] In some embodiments of this disclosure, the DPU-based data aggregation device 400 may further include a result feedback module.

[0129] The result feedback module can be used to generate a global column identifier corresponding to the global aggregation result after receiving the global aggregation result, store the global column identifier in shared memory, convert the global aggregation result from columnar to rowar format, and control the scheduling process in at least one process to send the rowar data corresponding to the global aggregation result to the user, or to send the rowar data corresponding to the global aggregation result to the next node, so that the next node can perform corresponding operations based on the rowar data corresponding to the global aggregation result.

[0130] It should be noted that, Figure 4 The DPU-based data aggregation device 400 shown can execute the various steps in the above method embodiments and realize the various processes and effects in the above method embodiments, which will not be elaborated here.

[0131] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0132] In this embodiment of the disclosure, Figure 5 The electronic devices shown can be servers or terminals, and terminals specifically include mobile phones, computers, or tablets, etc., without limitation.

[0133] like Figure 5 As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0134] Specifically, the processor 510 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this disclosure.

[0135] Memory 520 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 520 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway device. In a particular embodiment, memory 520 is a non-volatile solid-state memory. In a particular embodiment, memory 520 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0136] The processor 510 reads and executes computer program instructions stored in the memory 520 to perform the steps of the DPU-based data aggregation method provided in this embodiment of the disclosure.

[0137] In one example, the electronic device may also include a transceiver 530 and a bus 540. Wherein, as... Figure 5 As shown, the processor 510, memory 520 and transceiver 530 are connected via bus 540 and communicate with each other.

[0138] Bus 540 may include hardware, software, or both. For example, and not limited to, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 540 may include one or more buses.

[0139] This disclosure also provides a computer-readable storage medium that can store a computer program that, when executed by a processor, enables the processor to implement the DPU-based data aggregation method provided in this disclosure.

[0140] The aforementioned storage medium may, for example, include a memory 520 containing computer program instructions, which can be executed by a processor 510 of an electronic device to complete the DPU-based data aggregation method provided in this embodiment. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device.

[0141] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data aggregation method based on DPU, characterized in that, include: Obtain the target query statement and the data to be processed corresponding to the target query statement, start at least one process, and determine the target process from the at least one process to perform the global aggregation operation corresponding to the target query statement; For each process, the target column data corresponding to the process is sent to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result; Receive the partial aggregation result returned by the data processor, generate the result column identifier corresponding to the partial aggregation result, and store the result column identifier in the shared memory of the preset database; When it is determined that the local aggregation operations corresponding to at least one process have been completed, the target process is controlled to obtain the result column identifier corresponding to each process from the shared memory, determine the result column data corresponding to the result column identifier, and send the result column data to the data processor so that the data processor executes the global aggregation operation and returns the global aggregation result corresponding to the global aggregation operation. Receive the global aggregation result.

2. The method according to claim 1, characterized in that, The at least one process includes at least one worker process and one scheduler process. Determining the target process from the at least one process for executing the global aggregation operation corresponding to the target query statement includes: Based on the configuration parameters corresponding to the scheduling process, determine whether the scheduling process has the permission to perform global aggregation operations; If the scheduling process has permission to perform global aggregation operations, then the scheduling process is determined as the target process; If the scheduling process does not have permission to perform global aggregation operations, the target process for performing global aggregation operations is determined from the at least one worker process based on the changes in the preset atomic variables in the shared memory.

3. The method according to claim 1, characterized in that, Obtaining the data to be processed corresponding to the target query statement includes: Perform a data scanning operation corresponding to the target query statement, read the data corresponding to the target query statement, and determine the data as the data to be processed; The data to be processed is converted from row format to column format to obtain the column data corresponding to the data to be processed and the column identifier corresponding to each column data.

4. The method according to claim 1, characterized in that, For each process, sending the target column data corresponding to that process to the data processor includes: The target column identifier corresponding to the process is determined, and the target column identifier is sent to the accelerated computing engine by calling the first preset interface based on the preset target plugin, so that the accelerated computing engine determines the target column data corresponding to the target column identifier based on the mapping relationship between the column identifier and the column data; The target column data is sent to the data processor by calling the second preset interface based on the accelerated computing engine.

5. The method according to claim 1, characterized in that, The step of receiving the local aggregation result returned by the data processor, generating the result column identifier corresponding to the local aggregation result, and storing the result column identifier in the shared memory of a preset database includes: The accelerated computing engine receives the local aggregation results, generates the result column identifier corresponding to the local aggregation results, and sends the result column identifier to the target plugin. The result column identifier is stored in the shared memory based on the target plugin.

6. The method according to claim 1, characterized in that, After storing the result column identifier in the shared memory of a preset database, the method further includes: Based on the target plugin, the flag bit corresponding to the process is modified to obtain the target flag bit, which is used to indicate that the local aggregation operation corresponding to the process has been completed.

7. The method according to claim 1, characterized in that, After receiving the global aggregation result, the method further includes: Generate a global column identifier corresponding to the global aggregation result, and store the global column identifier in the shared memory; The global aggregation result is converted from columnar to rowar format. The scheduling process in at least one process is controlled to send the rowar data corresponding to the global aggregation result to the user, or to send the rowar data corresponding to the global aggregation result to the next node, so that the next node can perform corresponding operations based on the rowar data corresponding to the global aggregation result.

8. A data aggregation device based on a DPU, characterized in that, include: The data acquisition module is used to acquire the target query statement and the data to be processed corresponding to the target query statement, start at least one process, and determine the target process from the at least one process to perform the global aggregation operation corresponding to the target query statement; The local aggregation module is used to send the target column data corresponding to each process to the data processor, so that the data processor can perform the local aggregation operation corresponding to the target column data, obtain the local aggregation result, and return the local aggregation result; The first receiving module is used to receive the local aggregation result returned by the data processor, generate the result column identifier corresponding to the local aggregation result, and store the result column identifier in the shared memory of the preset database. A global aggregation module is used to control the target process to obtain the result column identifier corresponding to each process from the shared memory when it is determined that the local aggregation operations corresponding to the at least one process have been completed, and to determine the result column data corresponding to the result column identifier, and to send the result column data to the data processor so that the data processor executes the global aggregation operation and returns the global aggregation result corresponding to the global aggregation operation. The second receiving module is used to receive the global aggregation result.

9. An electronic device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the DPU-based data aggregation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, causes the processor to implement the DPU-based data aggregation method as described in any one of claims 1-7.

Images