Data aggregation method and device based on DPU, equipment and medium
By adopting the DPU-based data aggregation method in the PostgreSQL database, the aggregation operation is offloaded to the data processor for execution, solving the problems of degraded performance and low efficiency of the central processor, and achieving efficient data aggregation processing.
Patent Information
- Application Number
- CN202411114783.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-14
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-14
AI Technical Summary
The aggregation operations of PostgreSQL databases when executing SQL queries are performed on the central processor, resulting in high data transmission overhead, degradation in performance and low efficiency.
Using DPU-based data aggregation method, local aggregation operations and global aggregation operations are offloaded to the data processor for execution, and directly access the result column identifier through shared memory, reducing data transmission, improving central processor performance and aggregation efficiency.
By offloading the aggregation operation to the data processor for execution, the data transmission overhead is reduced, the central processor performance and aggregation efficiency is improved, and it is suitable for environments that process large amounts of data.
Smart Images

Figure CN118981480A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of big data technology, and in particular to a DPU-based data aggregation method, device, equipment and medium. Background Art
[0002] PostgreSQL is an open source relational database management system. After receiving an SQL query statement, PostgreSQL starts multiple worker processes (such as workerprocesses). When an aggregation operation is involved in the execution of an SQL query statement, PostgreSQL's native aggregation node, namely the aggregate node, divides the to-be-processed data corresponding to the SQL query statement into multiple smaller chunks and distributes them to each worker process, so that multiple worker processes can perform local aggregation operations in parallel. After each worker process completes the local aggregation operation, the calculation result is passed to the query scheduling process. The query scheduling process performs a final global aggregation on the calculation results of the local aggregation operations returned by all worker processes, thereby obtaining the final aggregation result.
[0003] However, the entire aggregation operation is performed on the central processing unit, and there is data transmission overhead when each worker process passes the calculation result of the local aggregation operation to the query scheduling process, resulting in the problem of reduced central processing unit performance and low aggregation efficiency. Summary of the invention
[0004] In order to solve the above technical problems, the present disclosure provides a DPU-based data aggregation method, device, equipment and medium.
[0005] A first aspect of an embodiment of the present disclosure provides a DPU-based data aggregation method, including:
[0006] Obtain a target query statement and to-be-processed data corresponding to the target query statement, start at least one process, and determine a target process for executing a global aggregation operation corresponding to the target query statement from the at least one process;
[0007] For each process, the target column data corresponding to the process is sent to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result;
[0008] Receive the local aggregation result returned by the data processor, generate a result column identifier corresponding to the local aggregation result, and store the result column identifier in the shared memory of the preset database;
[0009] When it is determined that the local aggregation operations corresponding to at least one process are all executed, control the target process to obtain the result column identifier corresponding to each process from the shared memory, determine the result column data corresponding to the result column identifier, and send the result column data to the data processor, so that the data processor performs the global aggregation operation and returns the global aggregation result corresponding to the global aggregation operation;
[0010] Receive global aggregation results.
[0011] A second aspect of an embodiment of the present disclosure provides a DPU-based data aggregation device, including:
[0012] A data acquisition module, used to acquire a target query statement and the to-be-processed data corresponding to the target query statement, start at least one process, and determine a target process for executing a global aggregation operation corresponding to the target query statement from the at least one process;
[0013] A local aggregation module is used to send the target column data corresponding to each process to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result;
[0014] A first receiving module is used to receive the local aggregation result returned by the data processor, generate a result column identifier corresponding to the local aggregation result, and store the result column identifier in a shared memory of a preset database;
[0015] A global aggregation module is used to control the target process to obtain the result column identifier corresponding to each process from the shared memory when it is determined that the local aggregation operations corresponding to at least one process have been executed, and to determine the result column data corresponding to the result column identifier, and to send the result column data to the data processor, so that the data processor performs the global aggregation operation and returns the global aggregation result corresponding to the global aggregation operation;
[0016] The second receiving module is used to receive the global aggregation result.
[0017] A third aspect of the embodiments of the present disclosure provides an electronic device, including:
[0018] processor;
[0019] A memory for storing executable instructions;
[0020] The processor is used to read executable instructions from the memory and execute the executable instructions to implement the DPU-based data aggregation method provided in the first aspect above.
[0021] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the DPU-based data aggregation method provided in the first aspect.
[0022] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0023] The DPU-based data aggregation method, device, equipment and medium provided by the embodiments of the present disclosure can obtain a target query statement and the data to be processed corresponding to the target query statement, start at least one process, and determine a target process for executing the global aggregation operation corresponding to the target query statement from the at least one process. For each process, the target column data corresponding to the process is sent to the data processor, so that the data processor executes the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result. The local aggregation result returned by the data processor is received, and a result column identifier corresponding to the local aggregation result is generated, and the result column identifier is stored in the shared memory of the preset database. When it is determined that the local aggregation operations corresponding to at least one process have been executed, the target process is controlled to obtain the local aggregation result from the shared memory. The result column identifier corresponding to each process is obtained, and the result column data corresponding to the result column identifier is determined, and the result column data is sent to the data processor, so that the data processor performs a global aggregation operation, and returns a global aggregation result corresponding to the global aggregation operation, and receives the global aggregation result. Thus, both the local aggregation operation and the global aggregation operation can be offloaded to the data processor for execution, and after the local aggregation result is obtained, the result column identifier corresponding to the local aggregation result is stored in the shared memory of the preset database, and the target process that performs the global aggregation operation can directly access the result column identifier in the shared memory, and other processes do not need to transmit the local aggregation result to the target process through data transmission after executing the local aggregation operation, thereby reducing data transmission overhead and improving central processing unit performance and aggregation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0026] Figure 1 is a flow chart of a DPU-based data aggregation method provided by an embodiment of the present disclosure;
[0027] Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;
[0028] Figure 3 is a flow chart of another DPU-based data aggregation method provided by an embodiment of the present disclosure;
[0029] Figure 4 is a structural schematic diagram of a DPU-based data aggregation device provided in an embodiment of the present disclosure;
[0030] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0033] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0034] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0035] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0036] Generally, the aggregation operations corresponding to existing query statements are all executed on the central processing unit, and when each worker process passes the calculation results of the local aggregation operation to the query scheduling process, there is data transmission overhead, which leads to the performance degradation of the central processing unit and low aggregation efficiency. To address this problem, the embodiment of the present disclosure provides a data aggregation method based on DPU, which is introduced below in conjunction with specific embodiments.
[0037] Figure 1 is a flow chart of a DPU-based data aggregation method provided by an embodiment of the present disclosure. The method can be executed by a DPU-based data aggregation device. The DPU-based data aggregation device can be implemented in software and / or hardware. The DPU-based data aggregation device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc. In addition, the method can be applied to Figure 2 The application scenario shown includes an electronic device 21 and a data processor 22. It can be understood that the DPU-based data aggregation method provided in the embodiment of the present disclosure can also be applied in other scenarios.
[0038] like Figure 2 As shown, the electronic device 21 includes a preset database, a target plug-in, a shared memory of the preset database, an accelerated computing engine, and a worker process 1, a worker process 2, and a scheduling process started based on the target query statement and the preset database.
[0039] The preset database may be a PostgreSQL database, which is an open source relational database management system, or may be other SQL databases, which are not limited here.
[0040] The target plug-in can be pg-race, which is an extension of the PostgreSQL database. It aims to use the data processor to accelerate batch processing and analysis workloads. In the embodiment of the present disclosure, it can be based on the target plug-in to achieve interaction with the data processor (Data Processor Unit, DPU), and use the data processor to perform data aggregation processing to improve the efficiency of data aggregation processing and reduce the data processing pressure of the central processing unit.
[0041] In an embodiment of the present disclosure, the target plug-in can be used to call a preset interface to send a column identifier to an accelerated computing engine, where the column identifier can be understood as a column ID; it can also be used to store the column identifier, the result column identifier corresponding to the local aggregation result, and the global column identifier corresponding to the global aggregation result in the shared memory of the preset database.
[0042] The shared memory of the preset database can be understood as a memory area shared by all background processes in the preset database, which is used to store various data structures that need to be shared between multiple processes. When the preset database is a PostgreSQL database, the shared memory of the preset database is the shared memory of the PostgreSQL database.
[0043] In an embodiment of the present disclosure, the shared memory of the preset database can be used to store column identifiers, including column identifiers corresponding to the data to be processed, result column identifiers corresponding to the local aggregation results, global column identifiers corresponding to the global aggregation results, etc., for calling by each process.
[0044] The accelerated computing engine can be used to provide an interface for accessing column data, and can obtain column data corresponding to the column identifier based on the column identifier query sent by the target plug-in; it can also call the data processor based on the preset interface to enable the data processor to perform local aggregation operations and / or global aggregation operations corresponding to the target query statement. For example, the accelerated computing engine can be a Rendering And Computing Engine, abbreviated as RACE.
[0045] Worker process 1, worker process 2 and scheduling process are processes started after receiving a target query statement and used to perform an aggregation operation of the target query statement, wherein the aggregation operation includes a local aggregation operation and a global aggregation operation.
[0046] In the embodiment of the present disclosure, the electronic device 21 and the data processor 22 may be connected via a slot connection, a network communication connection, or the like.
[0047] Combine the following Figure 2 The application scenario shown is Figure 1 The DPU-based data aggregation method is introduced as shown, for example, Figure 2 The electronic device 21 in the embodiment can execute the method. Figure 1 As shown, the DPU-based data aggregation method provided in this embodiment includes the following steps.
[0048] S110: Obtain a target query statement and data to be processed corresponding to the target query statement, start at least one process, and determine a target process for executing a global aggregation operation corresponding to the target query statement from the at least one process.
[0049] In the disclosed embodiment, the target query statement may be a statement for retrieving or querying data from a database, wherein the specific syntax of the target query statement varies based on the database management system used. For example, the target query statement may be a structured query language (SQL).
[0050] In an embodiment of the present disclosure, the target query statement may be a query statement involving data aggregation, wherein data aggregation may refer to a process of merging data from multiple data sources or different data from the same data source into a single data set.
[0051] Exemplarily, the target query statement includes an aggregation function, and the aggregation function may include a sum function (SUM), an average function (AVG), a maximum value function (MAX), a minimum value function (MIN), and the like.
[0052] The data to be processed can be understood as data required for executing the target query statement, wherein the acquired data to be processed is row-based data.
[0053] A process can be understood as the process of executing the target query statement.
[0054] The specific implementation method of starting at least one process is similar to the existing implementation method of starting a process during the execution of an SQL query statement, and is not described in detail here.
[0055] Specifically, the electronic device can receive the query request sent by the user in real time, parse the query request to obtain the target query statement, and after obtaining the target query statement, read the to-be-processed data corresponding to the target query statement from one or more of the memory, disk, preset database, etc.
[0056] S120. For each process, the target column data corresponding to the process is sent to the data processor, so that the data processor performs a local aggregation operation corresponding to the target column data, obtains a local aggregation result, and returns the local aggregation result.
[0057] In the embodiments of the present disclosure, the local aggregation operation can be understood as a partial aggregation operation, that is, a process of performing partial aggregation calculation on the divided data, wherein the division can be performed based on one or more attributes of the data.
[0058] Specifically, after obtaining the data to be processed corresponding to the target query statement, the electronic device performs block processing on the data to be processed, determines the block data corresponding to each process, and performs row-to-column processing on each block data to obtain the target column data corresponding to each block data, and sends the target column data corresponding to each process to the data processor, so that the data processor executes the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result. Since the data format supported by the data processor is columnar, the obtained local aggregation result is also in columnar form.
[0059] S130. Receive the local aggregation result returned by the data processor, generate a result column identifier corresponding to the local aggregation result, and store the result column identifier in the shared memory of the preset database.
[0060] In the embodiment of the present disclosure, the result column identifier can be understood as the column ID corresponding to the local aggregation result, that is, the unique identification information used to characterize the local aggregation result.
[0061] Specifically, after receiving the local aggregation result returned by the data processor, the electronic device generates a result column identifier corresponding to the local aggregation result based on a preset identifier generation method, and stores the result column identifier in the shared memory of the preset database.
[0062] S140. When it is determined that the local aggregation operations corresponding to at least one process have been executed, control the target process to obtain the result column identifier corresponding to each process from the shared memory, and determine the result column data corresponding to the result column identifier, and send the result column data to the data processor, so that the data processor performs a global aggregation operation and returns a global aggregation result corresponding to the global aggregation operation.
[0063] In the embodiments of the present disclosure, the global aggregation operation may be understood as a process of further summarizing and calculating the local aggregation results of the local aggregation operation, ie, the intermediate result set, after the local aggregation operation.
[0064] For example, the original aggregation function can be decomposed as follows:
[0065] min(x)=fmin(pmin(x))
[0066] max(x)=fmax(pmax(x))
[0067] sum(x)=fsum(psum(x))
[0068] count(x)=fsum(pcount(x))
[0069] avg(x)=fsum(psum()) / fsum(pcount(x))
[0070] Among them, the function starting with f represents the global aggregation operation, namely the final aggregate operation; the function starting with p represents the local aggregation operation, namely the partial aggregate operation.
[0071] Specifically, after receiving the local aggregation results of the target column data corresponding to each process returned by the data processor, and / or responding to the notification message returned by each process to indicate the completion of the execution, the electronic device determines that the local aggregation operations corresponding to at least one process have been completed, controls the target process to obtain the result column identifier corresponding to each process from the shared memory, and determines the result column data corresponding to the result column identifier, and sends the result column data to the data processor, so that the data processor performs a global aggregation operation based on the result column data, and returns the global aggregation result corresponding to the global aggregation operation to the electronic device. Since the data format supported by the data processor is columnar, the global aggregation result obtained is also in columnar form.
[0072] S150: Receive global aggregation results.
[0073] In the embodiment of the present disclosure, the electronic device may receive the global aggregation result returned by the data processor in real time.
[0074] In an embodiment of the present disclosure, it is possible to obtain a target query statement and data to be processed corresponding to the target query statement, start at least one process, and determine a target process for executing a global aggregation operation corresponding to the target query statement from at least one process. For each process, the target column data corresponding to the process is sent to a data processor, so that the data processor executes a local aggregation operation corresponding to the target column data, obtains a local aggregation result, and returns the local aggregation result. The local aggregation result returned by the data processor is received, and a result column identifier corresponding to the local aggregation result is generated, and the result column identifier is stored in a shared memory of a preset database. When it is determined that the local aggregation operations corresponding to at least one process have been executed, the target process is controlled to obtain the result column corresponding to each process from the shared memory. An identifier is used to determine the result column data corresponding to the result column identifier, and the result column data is sent to the data processor so that the data processor performs a global aggregation operation, and returns a global aggregation result corresponding to the global aggregation operation, and receives the global aggregation result. Thus, both the local aggregation operation and the global aggregation operation can be offloaded to the data processor for execution, and after the local aggregation result is obtained, the result column identifier corresponding to the local aggregation result is stored in the shared memory of the preset database, and the target process that performs the global aggregation operation can directly access the result column identifier in the shared memory, without the need for other processes to transmit the local aggregation result to the target process through data transmission after the local aggregation operation is performed, thereby reducing data transmission overhead and improving central processing unit performance and aggregation efficiency.
[0075] Based on the above-mentioned real-time example of the present invention, the calculation process of data aggregation can be executed on the data processor, which improves the processing efficiency of data aggregation and reduces the occupation of central processing unit and memory resources. It is very beneficial for processing large amounts of data and environments with limited resources, and can also improve the processing efficiency of large amounts of data.
[0076] Based on the above embodiments of the present disclosure, at least one process includes at least one worker process and one scheduling process, such as Figure 2 Worker process 1, worker process 2 and scheduling process shown in .
[0077] In an embodiment of the present disclosure, determining a target process for executing a global aggregation operation corresponding to a target query statement from at least one process may specifically include: determining whether a scheduling process has permission to execute the global aggregation operation based on configuration parameters corresponding to the scheduling process; if the scheduling process has permission to execute the global aggregation operation, determining the scheduling process as the target process; if the scheduling process does not have permission to execute the global aggregation operation, determining the target process for executing the global aggregation operation from at least one worker process based on the change value of an atomic variable preset in a shared memory.
[0078] In the embodiments of the present disclosure, configuration parameters may be understood as parameters required for setting the scheduling process to perform aggregation operations (including global aggregation operations and / or local aggregation operations), including resource parameters, permission parameters and other information.
[0079] Specifically, after starting the scheduling process, the electronic device obtains the configuration parameters of the scheduling process and determines whether the scheduling process has the authority to perform global aggregation operations based on the configuration parameters. If it is determined that the scheduling process has the authority to perform global aggregation operations, the scheduling process is determined as the target process; otherwise, a worker process is selected from at least one worker process and determined as the target process to perform the global aggregation operation.
[0080] Among them, the specific method of selecting a worker process from at least one worker process and determining it as the target process is as follows: an atomic variable is pre-set in the shared memory, wherein only one process (such as the worker process) can modify the value of the atomic variable at the same time, the initial value of the atomic variable is 0, and an addition operation is performed on the value of the atomic variable when each worker process is started, and then the worker process whose value of the atomic variable changes from 0 to 1 is determined as the target process for performing the global aggregation operation.
[0081] Exemplarily, the number of worker processes is 3. When the first worker process is started, the value of the atomic variable changes from 0 to 1. When the second worker process is started, the value of the atomic variable changes from 1 to 2. When the third worker process is started, the value of the atomic variable changes from 1 to 2. At this time, the first worker process changes the value of the atomic variable from 0 to 1, so the first worker process is determined as the target process for performing the global aggregation operation.
[0082] In the disclosed embodiment, when the scheduling process does not have the permission to execute the global aggregation operation, a target process for executing the global aggregation operation can be selected from at least one worker process, thereby improving the flexibility of process configuration and the flexibility of executing the global aggregation operation.
[0083] In an embodiment of the present disclosure, obtaining the data to be processed corresponding to the target query statement may specifically include: executing a data scanning operation corresponding to the target query statement, reading the data corresponding to the target query statement, and determining the data as the data to be processed; converting the data to be processed from row format to column format, obtaining the column format data corresponding to the data to be processed and the column identifier corresponding to each column of data.
[0084] Specifically, after obtaining the target query statement, the electronic device can parse the target query statement to ensure that the target query statement meets the preset standard (for example, when the database is a PostgreSQL database, ensure that the target query statement meets the SQL standard of the PostgreSQL database); after determining that the preset standard is met, perform a semantic analysis operation to check whether the semantics of the target query statement is correct, and ensure that the query elements exist and meet the semantic rules of the preset database; if it is determined that it meets the standard, perform a query rewrite operation to optimize the query structure of the target query statement, and generate multiple possible query execution plans, and select the execution plan with the lowest cost for execution, wherein the execution plan includes multiple nodes, each node represents a specific operation, such as scanning a table, i.e., a scanning operation, connecting a table, or aggregating data, i.e., an aggregation operation (including at least one of a local aggregation operation and a global aggregation operation); after determining the execution plan with the lowest cost, perform a data scanning operation corresponding to the target query statement, read data corresponding to the target query statement from at least one of a memory, a disk, or a preset database, determine the data as data to be processed, and convert the data to be processed from row format to column format to obtain column format data corresponding to the data to be processed and a column identifier corresponding to each column of data.
[0085] In the embodiment of the present disclosure, after obtaining the column identifier corresponding to each column of data, a mapping relationship between the column data and the column identifier is established.
[0086] In the embodiments of the present disclosure, after obtaining the data to be processed, row-to-column conversion processing can be performed on the data to be processed, so that the obtained columnar data can adapt to the data format of the data processor, and at the same time, a column identifier corresponding to the columnar data is generated and stored in a shared memory, so that the column data can be directly called from the shared memory when it is subsequently used, thereby reducing data transmission overhead.
[0087] Furthermore, for each process, the target column data corresponding to the process is sent to the data processor, which may specifically include: determining the target column identifier corresponding to the process, calling the first preset interface based on the preset target plug-in to send the target column identifier to the accelerated computing engine, so that the accelerated computing engine determines the target column data corresponding to the target column identifier based on the mapping relationship between the column identifier and the column data; calling the second preset interface based on the accelerated computing engine to send the target column data to the data processor.
[0088] In an embodiment of the present disclosure, the target plug-in can be used to store column identifiers in shared memory, and an interface for calling an accelerated computing engine is provided for calling the accelerated computing engine; the accelerated computing engine can be used to obtain column data, and can also call a data processor so that the data processor processes an aggregation operation (including global aggregation operations and local aggregation operations) of the column data.
[0089] In the embodiment of the present disclosure, the accelerated computing engine may send the operator corresponding to the target column data to the data processor at the same time as sending the target column data to the data processor.
[0090] In the embodiment of the present disclosure, the column data corresponding to each process can be sent to the data processor through the target plug-in and the accelerated computing engine, so that the data processor performs data aggregation processing.
[0091] In an embodiment of the present disclosure, a local aggregation result returned by a data processor is received, and a result column identifier corresponding to the local aggregation result is generated, and the result column identifier is stored in a shared memory of a preset database. Specifically, it may include: receiving the local aggregation result based on an accelerated computing engine, and generating a result column identifier corresponding to the local aggregation result, and sending the result column identifier to a target plug-in; storing the result column identifier in a shared memory based on the target plug-in.
[0092] In the embodiment of the present disclosure, after the result column identifier corresponding to the local aggregation result is generated, a mapping relationship between the local aggregation result and the result column identifier is established.
[0093] In the embodiment of the present disclosure, the result column identifier can be stored in the shared memory, so as to facilitate the subsequent calling of the local aggregation result based on the result column identifier in the shared memory.
[0094] Furthermore, after storing the result column identifier in the shared memory of the preset database, the DPU-based data aggregation method may also include: performing a modification operation on the flag bit corresponding to the process based on the target plug-in to obtain a target flag bit, and the target flag bit is used to indicate that the local aggregation operation corresponding to the process has been executed.
[0095] In some embodiments of the present disclosure, after storing the result column identifier corresponding to each process in the shared memory, the electronic device can modify the flag bit corresponding to each process based on the target plug-in, and modify the first flag bit corresponding to the original execution state to the second flag bit corresponding to the execution completion state, that is, the target flag bit, so as to notify the scheduling process that the local aggregation operation of the process has been completed.
[0096] In some other embodiments of the present disclosure, the electronic device can control each process to send NULL data to the scheduling process after storing the result column identifier corresponding to each process in the shared memory, indicating that the process will no longer transmit valid data to the gather node corresponding to the scheduling process (such as Figure 2 Node 1 in , so that the gather node transmits the local aggregation results to the target process to perform the final global aggregation operation.
[0097] In the disclosed embodiment, the scheduling process can be notified of the completion of the local aggregation operation by modifying the process flag, so that the scheduling process performs subsequent operations in response to the modification of the flag, thereby improving the effectiveness and efficiency of the aggregation operation.
[0098] In the embodiment of the present disclosure, the specific implementation method of determining the result column data corresponding to the result column identifier and sending the result column data to the data processor is similar to the specific implementation method of sending the target column data corresponding to each process to the data processor, and will not be repeated here.
[0099] In an embodiment of the present disclosure, after receiving the global aggregation result, the DPU-based data aggregation method may further include: generating a global column identifier corresponding to the global aggregation result, and storing the global column identifier in a shared memory; converting the global aggregation result from column to row format, and controlling a scheduling process in at least one process to send the row data corresponding to the global aggregation result to a user, or transmitting the row data corresponding to the global aggregation result to the next node, so that the next node performs corresponding operations based on the row data corresponding to the global aggregation result.
[0100] Specifically, the acceleration computing engine receives the global aggregation result, generates a global column identifier corresponding to the global aggregation result, sends the global column identifier to the target plug-in, stores the global column identifier in the shared memory based on the target plug-in, and converts the global aggregation result from column to row based on the target plug-in. The scheduling process in at least one process is controlled to send the row data corresponding to the global aggregation result to the user, or, when there is still a next execution node after the aggregation operation, transmits the row data corresponding to the global aggregation result to the next node, so that the next node executes the operation corresponding to the next node based on the row data corresponding to the global aggregation result.
[0101] In the embodiments of the present disclosure, after obtaining the global aggregation result, it can be fed back to the user in a timely manner or sent to the next node, and the next node executes the subsequent operations, thereby improving the user experience and the timeliness of the execution operation of the next node. Since the output format of the global aggregation result obtained by the DPU-based data aggregation operation is consistent with the output format of the native aggregate node of the preset database, it will not affect the native nodes of the preset database, thereby enhancing the compatibility and portability of the data processor in the process of executing data aggregation.
[0102] Figure 3 is a flow chart of another DPU-based data aggregation method provided by an embodiment of the present disclosure, such as Figure 3 As shown, the DPU-based data aggregation method may specifically include the following steps:
[0103] S310, obtaining a target query statement, executing a data scanning operation corresponding to the target query statement, reading data corresponding to the target query statement, determining the data as data to be processed, converting the data to be processed from row format to column format, obtaining column format data corresponding to the data to be processed and a column identifier corresponding to each column of data.
[0104] S320: Start at least one process, and determine a target process for executing a global aggregation operation corresponding to a target query statement from the at least one process.
[0105] S330. For each process, determine the target column identifier corresponding to the process, and based on the preset target plug-in, call the first preset interface to send the target column identifier to the accelerated computing engine, so that the accelerated computing engine determines the target column data corresponding to the target column identifier based on the mapping relationship between the column identifier and the column data.
[0106] S340, based on the accelerated computing engine, calling the second preset interface to send the target column data to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result.
[0107] S350, receiving the local aggregation result based on the accelerated computing engine, and generating a result column identifier corresponding to the local aggregation result, sending the result column identifier to the target plug-in, and storing the result column identifier in the shared memory based on the target plug-in.
[0108] S360. When it is determined that the local aggregation operations corresponding to at least one process have been executed, control the target process to obtain the result column identifier corresponding to each process from the shared memory, and determine the result column data corresponding to the result column identifier, and send the result column data to the data processor so that the data processor performs a global aggregation operation and returns a global aggregation result corresponding to the global aggregation operation.
[0109] S370, receive the global aggregation result, generate a global column identifier corresponding to the global aggregation result, store the global column identifier in the shared memory, convert the global aggregation result from column format to row format, control the scheduling process in at least one process to send the row data corresponding to the global aggregation result to the user, or transmit the row data corresponding to the global aggregation result to the next node, so that the next node performs corresponding operations based on the row data corresponding to the global aggregation result.
[0110] It should be noted that the specific implementation of steps S310-S370 is similar to the implementation of the relevant steps in the above-mentioned embodiment of the present disclosure, and will not be repeated here.
[0111] In the embodiments of the present disclosure, both local aggregation operations and global aggregation operations can be offloaded to a data processor for execution, and after obtaining the local aggregation results, the result column identifiers corresponding to the local aggregation results are stored in the shared memory of a preset database, and the target process executing the global aggregation operation can directly access the result column identifiers in the shared memory. There is no need for other processes to transmit the local aggregation results to the target process through data transmission after executing the local aggregation operation, which reduces data transmission overhead and improves central processing unit performance and aggregation efficiency.
[0112] Figure 4 It is a structural diagram of a DPU-based data aggregation device provided in an embodiment of the present disclosure.
[0113] In the embodiments of the present disclosure, the DPU-based data aggregation device can be set in an electronic device and is understood as a part of the functional modules in the above electronic device. Specifically, the electronic device can be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., which is not limited here.
[0114] like Figure 4 As shown, the DPU-based data aggregation device 400 may include a data acquisition module 410 , a local aggregation module 420 , a first receiving module 430 , a global aggregation module 440 and a second receiving module 450 .
[0115] The data acquisition module 410 may be used to acquire a target query statement and the to-be-processed data corresponding to the target query statement, start at least one process, and determine a target process for executing a global aggregation operation corresponding to the target query statement from the at least one process.
[0116] The local aggregation module 420 can be used to send the target column data corresponding to each process to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result.
[0117] The first receiving module 430 may be configured to receive the local aggregation result returned by the data processor, generate a result column identifier corresponding to the local aggregation result, and store the result column identifier in a shared memory of a preset database.
[0118] The global aggregation module 440 can be used to control the target process to obtain the result column identifier corresponding to each process from the shared memory when it is determined that the local aggregation operations corresponding to at least one process have been executed, and determine the result column data corresponding to the result column identifier, and send the result column data to the data processor so that the data processor performs the global aggregation operation and returns the global aggregation result corresponding to the global aggregation operation.
[0119] The second receiving module 450 may be configured to receive a global aggregation result.
[0120] In an embodiment of the present disclosure, it is possible to obtain a target query statement and data to be processed corresponding to the target query statement, start at least one process, and determine a target process for executing a global aggregation operation corresponding to the target query statement from at least one process. For each process, the target column data corresponding to the process is sent to a data processor, so that the data processor executes a local aggregation operation corresponding to the target column data, obtains a local aggregation result, and returns the local aggregation result. The local aggregation result returned by the data processor is received, and a result column identifier corresponding to the local aggregation result is generated, and the result column identifier is stored in a shared memory of a preset database. When it is determined that the local aggregation operations corresponding to at least one process have been executed, the target process is controlled to obtain the result column corresponding to each process from the shared memory. An identifier is used to determine the result column data corresponding to the result column identifier, and the result column data is sent to the data processor so that the data processor performs a global aggregation operation, and returns a global aggregation result corresponding to the global aggregation operation, and receives the global aggregation result. Thus, both the local aggregation operation and the global aggregation operation can be offloaded to the data processor for execution, and after the local aggregation result is obtained, the result column identifier corresponding to the local aggregation result is stored in the shared memory of the preset database, and the target process that performs the global aggregation operation can directly access the result column identifier in the shared memory, without the need for other processes to transmit the local aggregation result to the target process through data transmission after the local aggregation operation is performed, thereby reducing data transmission overhead and improving central processing unit performance and aggregation efficiency.
[0121] In some embodiments of the present disclosure, at least one process includes at least one worker process and one scheduling process.
[0122] The data acquisition module 410 can be specifically used to determine whether the scheduling process has the authority to perform global aggregation operations based on the configuration parameters corresponding to the scheduling process; if the scheduling process has the authority to perform global aggregation operations, the scheduling process is determined as the target process; if the scheduling process does not have the authority to perform global aggregation operations, the target process for performing global aggregation operations is determined from at least one worker process based on the change value of the atomic variable preset in the shared memory.
[0123] In some embodiments of the present disclosure, the data acquisition module 410 can also be specifically used to perform a data scanning operation corresponding to a target query statement, read the data corresponding to the target query statement, and determine the data as data to be processed; convert the data to be processed from row format to column format, and obtain the column format data corresponding to the data to be processed and the column identifier corresponding to each column of data.
[0124] In some embodiments of the present disclosure, the local aggregation module 420 can be specifically used to determine the target column identifier corresponding to the process, and call the first preset interface based on the preset target plug-in to send the target column identifier to the accelerated computing engine, so that the accelerated computing engine determines the target column data corresponding to the target column identifier based on the mapping relationship between the column identifier and the column data; and call the second preset interface based on the accelerated computing engine to send the target column data to the data processor.
[0125] In some embodiments of the present disclosure, the first receiving module 430 can be specifically used to receive local aggregation results based on the accelerated computing engine, generate a result column identifier corresponding to the local aggregation result, and send the result column identifier to the target plug-in; based on the target plug-in, store the result column identifier in a shared memory.
[0126] In some embodiments of the present disclosure, the DPU-based data aggregation device 400 may further include a flag modification module.
[0127] The flag modification module can be used to perform a modification operation on the flag corresponding to the process based on the target plug-in after storing the result column identifier in the shared memory of the preset database to obtain the target flag, which is used to indicate that the local aggregation operation corresponding to the process has been completed.
[0128] In some embodiments of the present disclosure, the DPU-based data aggregation device 400 may further include a result feedback module.
[0129] The result feedback module can be used to generate a global column identifier corresponding to the global aggregation result after receiving the global aggregation result, and store the global column identifier in the shared memory; convert the global aggregation result from column format to row format, and control the scheduling process in at least one process to send the row data corresponding to the global aggregation result to the user, or transmit the row data corresponding to the global aggregation result to the next node, so that the next node performs corresponding operations based on the row data corresponding to the global aggregation result.
[0130] It should be noted that Figure 4 The DPU-based data aggregation device 400 shown can execute each step in the above method embodiment and realize each process and effect in the above method embodiment, which will not be described in detail here.
[0131] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure.
[0132] In the disclosed embodiment, Figure 5 The electronic device shown may be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., which is not limited here.
[0133] like Figure 5 As shown, the electronic device may include a processor 510 and a memory 520 storing computer program instructions.
[0134] Specifically, the processor 510 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.
[0135] The memory 520 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 520 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 520 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 520 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 520 is a non-volatile solid-state memory. In a particular embodiment, the memory 520 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0136] The processor 510 reads and executes the computer program instructions stored in the memory 520 to perform the steps of the DPU-based data aggregation method provided in the embodiment of the present disclosure.
[0137] In one example, the electronic device may further include a transceiver 530 and a bus 540. Figure 5 As shown, the processor 510, the memory 520 and the transceiver 530 are connected via a bus 540 and communicate with each other.
[0138] The bus 540 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 540 may include one or more buses.
[0139] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the DPU-based data aggregation method provided by the embodiments of the present disclosure.
[0140] The above-mentioned storage medium may, for example, include a memory 520 of computer program instructions, and the above-mentioned instructions may be executed by the processor 510 of the electronic device to complete the DPU-based data aggregation method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0141] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data aggregation method based on DPU, characterized in that: include: Obtaining a target query statement and to-be-processed data corresponding to the target query statement, starting at least one process, and determining a target process for executing a global aggregation operation corresponding to the target query statement from the at least one process; For each process, the target column data corresponding to the process is sent to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result; Receive the local aggregation result returned by the data processor, generate a result column identifier corresponding to the local aggregation result, and store the result column identifier in a shared memory of a preset database; When it is determined that the local aggregation operations corresponding to the at least one process are all executed, controlling the target process to obtain the result column identifier corresponding to each process from the shared memory, and determining the result column data corresponding to the result column identifier, and sending the result column data to the data processor, so that the data processor performs the global aggregation operation and returns the global aggregation result corresponding to the global aggregation operation; The global aggregation result is received.
2. The method according to claim 1, characterized in that The at least one process includes at least one worker process and a scheduling process, and determining a target process for executing the global aggregation operation corresponding to the target query statement from the at least one process includes: Determining whether the scheduling process has permission to perform a global aggregation operation based on configuration parameters corresponding to the scheduling process; If the scheduling process has the permission to perform the global aggregation operation, determining the scheduling process as the target process; If the scheduling process does not have the authority to perform the global aggregation operation, a target process for performing the global aggregation operation is determined from the at least one worker process based on the change value of the atomic variable preset in the shared memory.
3. The method according to claim 1, characterized in that Obtaining the data to be processed corresponding to the target query statement, including: Execute a data scanning operation corresponding to the target query statement, read the data corresponding to the target query statement, and determine the data as the data to be processed; The data to be processed is converted from row format to column format to obtain column format data corresponding to the data to be processed and a column identifier corresponding to each column of data.
4. The method according to claim 1, characterized in that: For each process, sending the target column data corresponding to the process to the data processor includes: Determine a target column identifier corresponding to the process, and call a first preset interface based on a preset target plug-in to send the target column identifier to an accelerated computing engine, so that the accelerated computing engine determines the target column data corresponding to the target column identifier based on a mapping relationship between the column identifier and the column data; The target column data is sent to the data processor based on the accelerated computing engine calling a second preset interface.
5. The method according to claim 1, characterized in that The receiving the local aggregation result returned by the data processor, generating a result column identifier corresponding to the local aggregation result, and storing the result column identifier in a shared memory of a preset database includes: Receiving the local aggregation result based on the accelerated computing engine, generating a result column identifier corresponding to the local aggregation result, and sending the result column identifier to the target plug-in; The result column identifier is stored in the shared memory based on the target plug-in.
6. The method according to claim 1, characterized in that After storing the result column identifier in the shared memory of the preset database, the method further includes: A modification operation is performed on the flag bit corresponding to the process based on the target plug-in to obtain a target flag bit, where the target flag bit is used to indicate that the local aggregation operation corresponding to the process has been completed.
7. The method according to claim 1, characterized in that After receiving the global aggregation result, the method further includes: Generate a global column identifier corresponding to the global aggregation result, and store the global column identifier in the shared memory; Convert the global aggregation result from column format to row format, control the scheduling process in at least one process to send the row data corresponding to the global aggregation result to the user, or transmit the row data corresponding to the global aggregation result to the next node, so that the next node performs corresponding operations based on the row data corresponding to the global aggregation result.
8. A data aggregation device based on DPU, characterized in that: include: A data acquisition module, used to acquire a target query statement and the to-be-processed data corresponding to the target query statement, start at least one process, and determine a target process for executing the global aggregation operation corresponding to the target query statement from the at least one process; A local aggregation module is used to send the target column data corresponding to each process to the data processor, so that the data processor performs the local aggregation operation corresponding to the target column data, obtains the local aggregation result, and returns the local aggregation result; A first receiving module, used for receiving the local aggregation result returned by the data processor, generating a result column identifier corresponding to the local aggregation result, and storing the result column identifier in a shared memory of a preset database; A global aggregation module, configured to, when it is determined that all local aggregation operations corresponding to the at least one process have been executed, control the target process to obtain a result column identifier corresponding to each process from the shared memory, determine result column data corresponding to the result column identifier, and send the result column data to the data processor, so that the data processor executes the global aggregation operation and returns a global aggregation result corresponding to the global aggregation operation; The second receiving module is used to receive the global aggregation result.
9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the DPU-based data aggregation method described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the DPU-based data aggregation method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Service processing method and device based on cloud network access of tenants, equipment and medium
CN117376266A
Data processing method and device
CN117435330A
Aggregated query method and device based on distributed database and electronic equipment
CN117909369A
Response information generation method, system, device, equipment, medium and program product
CN118093847A
System, method, and program for performing aggregation process for each piece of received data
US20150120739A1
Cited By
Method and system for querying multi-source DPU flow table, DPU cluster and computing device
CN121255846A
Database processing method, device and storage medium
US20260050609A1
