Sql request processing method and device, and dpu based on kpu architecture
By introducing the DPU of the KPU architecture into Spark SQL, row and column transformation and data filtering are realized, which solves the problem of low CPU computing efficiency, improves the efficiency of SQL request processing, reduces CPU resource consumption, and improves computing reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YUSUR TECH CO LTD
- Filing Date
- 2025-05-26
- Publication Date
- 2026-07-21
AI Technical Summary
When Spark SQL performs computations on the CPU, there is redundant data processing and storage, which leads to low processing efficiency and increased CPU resource utilization, affecting the efficiency and reliability of computation scheduling.
SQL request processing is performed using a DPU based on a KPU architecture. Row-based databases are converted into columnar storage through row-column transformation. The DPU is used for data filtering and logical summation operations to reduce redundant data calculation and storage.
It improves the efficiency of SQL request processing, reduces CPU resource utilization, and enhances the reliability and efficiency of CPU computing scheduling.
Smart Images
Figure CN120849436B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to SQL request processing methods, apparatus, and DPU based on KPU architecture. Background Technology
[0002] Apache Spark is a fast, general-purpose computing engine designed for large-scale data processing, using an open-source cluster computing environment similar to Hadoop. Spark SQL is one of Spark's computing modules, specifically designed for processing structured data. Spark SQL allows users to execute SQL queries and read / write operations using standard SQL statements, as well as SQL statements from the data warehousing tool Hive to execute queries and read / write operations on Hive repositories.
[0003] Currently, Spark SQL is based on the Spark framework, and all its computations are still performed on the CPU. Besides maintaining the overall computational schedule, the CPU requires additional computing power for data-intensive computations specific to SQL requests. As a general-purpose processing chip, the CPU has no significant advantage in high-intensity big data computations. This makes the CPU's computing power the main bottleneck in SQL request processing performance due to redundant data computation and storage. This not only reduces SQL request processing efficiency but also increases CPU resource utilization, affecting the efficiency and reliability of CPU computational scheduling.
[0004] Therefore, there is an urgent need to design a method that can improve the processing efficiency of SQL request statements and reduce CPU resource utilization. Summary of the Invention
[0005] In view of this, embodiments of this application provide an SQL request processing method, apparatus, and a DPU based on a KPU architecture to eliminate or improve one or more defects existing in the prior art.
[0006] One aspect of this application provides an SQL request processing method, the method being executed by a data processor (DPU) based on a kernel processor (KPU) architecture, the method comprising:
[0007] The data tables in the row-based database are transformed into rows and columns to obtain attribute information rows corresponding to each attribute type. Each attribute information row contains all the fields of an attribute type in the data table.
[0008] Based on the attribute types corresponding to the judgment target and the returned object specified in the SQL request statement, the attribute information row corresponding to the judgment target and the attribute information row corresponding to the returned object are selected from each of the attribute information rows respectively, and the attribute information row corresponding to the judgment target is filtered by field based on the judgment conditions specified in the SQL request statement;
[0009] Based on the attribute information row corresponding to the target after field filtering, the attribute information row corresponding to the returned object is further filtered by field to obtain the request result data corresponding to the SQL request statement.
[0010] In some embodiments of this application, before performing row-to-column transformation on the data tables in the row-based database, the method further includes:
[0011] Based on the data reading operator preset in the DPU, a data table is read from a row-based database outside the DPU. Each column in the data table is used to store all fields corresponding to each attribute type, and each column in the data table corresponds one-to-one with each data type.
[0012] In some embodiments of this application, the step of performing row-to-column transformation on the data tables in the row-based database to obtain attribute information rows corresponding to each attribute type includes:
[0013] The row and column transformation operator preset in the DPU is invoked to perform row and column transformation on the data table to obtain the corresponding columnar storage data table. Each row in the columnar storage data table is a different attribute information row, and each attribute information row is used to store all fields corresponding to each attribute type. Each attribute information row in the columnar storage data table corresponds one-to-one with each data type.
[0014] In some embodiments of this application, after performing row-to-column transformation on the data tables in the row-based database, the method further includes:
[0015] It receives SQL request statements, reads the specified data table from the SQL request statement, determines the attribute type of the target, the judgment conditions, and the attribute type of the returned object.
[0016] In some embodiments of this application, the step of selecting the attribute information row corresponding to the judgment target and the attribute information row corresponding to the returned object from each of the attribute information rows according to the attribute types specified in the SQL request statement, and performing field filtering on the attribute information row corresponding to the judgment target based on the judgment conditions specified in the SQL request statement, includes:
[0017] The binary operation operator preset in the DPU is invoked to scan the columnar storage data table according to the attribute types corresponding to the judgment target and the return object specified in the SQL request statement, so as to obtain the attribute information row corresponding to the judgment target and the attribute information row corresponding to the return object, and to filter the attribute information row corresponding to the judgment target based on the judgment conditions specified in the SQL request statement.
[0018] In some embodiments of this application, before performing field filtering on the attribute information row corresponding to the returned object based on the attribute information row corresponding to the target after field filtering, the method further includes:
[0019] The binary operation operator is invoked to mask fields that meet the judgment conditions in the attribute information row corresponding to the judgment target using a first identifier, and fields that do not meet the judgment conditions using a second identifier, wherein the first identifier and the second identifier are different.
[0020] In some embodiments of this application, the step of filtering the attribute information row corresponding to the returned object based on the attribute information row corresponding to the target after field filtering, in order to obtain the request result data corresponding to the SQL request statement, includes:
[0021] The logical summation operator preset in the DPU is invoked. In the columnar storage data table, the first identifier in the attribute information row corresponding to the judgment target is logically summed with each field in the attribute information row corresponding to the returned object to obtain the filtered attribute information row corresponding to the returned object. The filtered attribute information row corresponding to the returned object is used as the request result data corresponding to the SQL request statement.
[0022] Output the request result data corresponding to the SQL request statement.
[0023] Another aspect of this application provides an SQL request processing apparatus, the apparatus being disposed in a data processor DPU based on a kernel processor KPU architecture, the apparatus comprising:
[0024] The row-column transformation module is used to transform the data table in the row-based database into rows and columns to obtain the attribute information rows corresponding to each attribute type. Each attribute information row contains all the fields of an attribute type in the data table.
[0025] The target filtering module is used to select the attribute information row corresponding to the target and the attribute information row corresponding to the returned object from each attribute information row according to the attribute types of the target and the returned object specified in the SQL request statement, and to perform field filtering on the attribute information row corresponding to the target based on the judgment conditions specified in the SQL request statement.
[0026] The object filtering module is used to filter the attribute information row corresponding to the returned object based on the attribute information row corresponding to the target after field filtering, so as to obtain the request result data corresponding to the SQL request statement.
[0027] In some embodiments of this application, the SQL request processing apparatus further includes:
[0028] The data reading module is used to read data tables from a row-based database outside the DPU based on a preset data reading operator in the DPU before performing row-column conversion on the data tables in the row-based database. Each column in the data table is used to store all fields corresponding to each attribute type, and each column in the data table corresponds one-to-one with each data type.
[0029] A third aspect of this application provides a DPU based on a KPU architecture, wherein the DPU includes an SQL request processing device for executing the SQL request processing method;
[0030] The SQL request processing device communicates with a row-based database outside the DPU; and the SQL request processing device communicates with an interactive query module in the distributed computing platform to receive SQL request statements from the interactive query module.
[0031] The SQL request processing method provided in this application is executed by a data processor (DPU) based on a kernel processor (KPU) architecture. The method transforms a data table in a row-based database into rows and columns to obtain attribute information rows corresponding to each attribute type. Each attribute information row contains all fields of an attribute type in the data table. Based on the attribute types corresponding to the judgment target and the return object specified in the SQL request statement, the method selects the attribute information rows corresponding to the judgment target and the return object from each of the attribute information rows. Then, based on the judgment conditions specified in the SQL request statement, the method filters the attribute information rows corresponding to the judgment target by fields. Finally, based on the field-filtered attribute information rows corresponding to the judgment target, the method filters the attribute information rows corresponding to the return object by fields to obtain the request result data corresponding to the SQL request statement. This method reduces the computation and storage of redundant data for the CPU when processing SQL request statements, thereby reducing CPU resource utilization, improving the reliability of CPU SQL retrieval, and effectively improving the processing efficiency of SQL request statements.
[0032] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.
[0033] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description
[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings:
[0035] Figure 1 This is a schematic diagram of the first type of SQL request processing method in one embodiment of this application.
[0036] Figure 2 This is a schematic diagram of a second flow of the SQL request processing method in one embodiment of this application.
[0037] Figure 3This is a schematic diagram of the third flow of the SQL request processing method in one embodiment of this application.
[0038] Figure 4 This is a schematic diagram of a first structure of an SQL request processing device in one embodiment of this application.
[0039] Figure 5 This is a schematic diagram of a second structure of the SQL request processing device in one embodiment of this application.
[0040] Figure 6 This is a schematic diagram of the structure of a DPU based on the KPU architecture in one embodiment of this application. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.
[0042] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.
[0043] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.
[0044] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.
[0045] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.
[0046] In one or more embodiments of this application, Spark is a distributed computing platform, a computing framework written in Scala, and a fast, general-purpose, and scalable big data analysis engine based on memory. Spark SQL is a module based on Spark for processing structured data. Spark SQL supports querying and loading from various data sources, is compatible with Hive, and can execute SQL statements using JDBC / ODBC connections. It provides important technical support for the Spark framework in structured data analysis.
[0047] To improve the processing efficiency of SQL request statements and reduce CPU resource utilization, embodiments of this application provide an SQL request processing method, an SQL request processing device for executing the SQL request processing method, and a DPU based on a KPU architecture. These methods can reduce the computation and storage of redundant data brought to the CPU by processing SQL request statements, thereby reducing CPU resource utilization, improving the reliability of the CPU in SQL retrieval, and effectively improving the processing efficiency of SQL request statements.
[0048] The following examples will provide a detailed description.
[0049] Based on this, embodiments of this application provide an SQL request processing method that can be implemented by an SQL request processing device, executed by a data processor DPU based on a kernel processor KPU architecture, see [link to relevant documentation]. Figure 1 The SQL request processing method specifically includes the following:
[0050] Step 100: Transform the data tables in the row-based database into rows and columns to obtain attribute information rows corresponding to each attribute type. Each attribute information row contains all the fields of an attribute type in the data table.
[0051] This is understandable. As a dedicated data processing chip, the DPU chip based on the KPU architecture offers a significant performance improvement over the CPU when handling complex data computations. Therefore, offloading Spark SQL data computation from the CPU to the DPU can greatly enhance Spark SQL performance, accelerate Spark SQL computation in big data scenarios, and allow the CPU to focus on Spark's computation scheduling while the DPU focuses on data computation within Spark SQL.
[0052] However, although DPUs have significant advantages over CPUs in performing data-intensive computations, their computing architecture is better suited for columnar operations. Some SQL statements are more advantageous for columnar operations than row-based operations. In other words, when performing SQL operations, traditional CPUs, because data storage and retrieval are row-based, need to read data from each row for calculation, which greatly reduces the efficiency of the operation and requires additional storage space for data storage.
[0053] Based on this, in one or more embodiments of this application, a columnar data calculation method based on DPU is proposed, and the basis for executing the columnar data is the above-mentioned step 100, which requires row and column transformation of the data table in the row database.
[0054] It is understood that the data table in the row-based database refers to a row-based data table, in which each row represents an independent record, and each record contains a field corresponding to each attribute type. Each column is used to store all fields corresponding to each attribute type, and each column in the data table corresponds one-to-one with each of the data types.
[0055] In one example of this application, the rows of the data table refer to all rows except the first row, which is used only to represent the various data types. As shown in Table 1, the row-based data table can contain three attribute types: "name", "age", and "score".
[0056] Table 1 is a row-based data table.
[0057] Name age Fraction Zhang San 21 95 Li Si 19 59
[0058] After performing row and column transformation on the row-based data table, the resulting attribute information rows for each attribute type are as follows: Each attribute information row contains all the fields of one attribute type in the data table. Based on Table 1 above, the attribute information rows for the three attribute types "Name," "Age," and "Score" can each contain:
[0059] (1) The attribute information row corresponding to the attribute type "Name": "Zhang San, Li Si";
[0060] (2) The attribute information row corresponding to the attribute type "Age": "21,19";
[0061] (3) The attribute information row corresponding to the attribute type "score": "95,59".
[0062] Step 200: Based on the attribute types corresponding to the judgment target and the returned object specified in the SQL request statement, select the attribute information row corresponding to the judgment target and the attribute information row corresponding to the returned object from each of the attribute information rows, and perform field filtering on the attribute information row corresponding to the judgment target based on the judgment conditions specified in the SQL request statement.
[0063] It is understood that the SQL request statement refers to an SQL statement containing an operation type, which may include SELECT, etc.; and each SQL request statement contains: operation type, data table in row-based database, attribute type corresponding to the judgment target, judgment condition, and attribute type corresponding to the return object. The judgment target refers to the attribute used as the judgment object in the judgment condition, and the return object specifies the attribute to be returned. In a special case, the attribute corresponding to the target and the attribute corresponding to the return object may be the same, but usually they are different.
[0064] Based on Table 1 above, in one example, the SQL request statement can be:
[0065] "SELECT name FROM TABLE 1 WHERE score>60";
[0066] In the above SQL request statement, the operation type is "SELECT"; the data table in the row database is "TABLE1"; the attribute type of the target is "name"; the judgment condition is "score > 60"; and the attribute type of the returned object is "score".
[0067] This SQL statement means: select the names of students with scores greater than 60 from the table. In the example above, it will return: Zhang San.
[0068] When the CPU executes this statement, because the data is calculated row-wise, the CPU reads all the data in the first row: Zhang San, 21, 95, and then checks if 95 > 60. Therefore, it stores the name Zhang San as the result in the result set. Next, the CPU reads all the data in the second row: Li Si, 19, 59, and checks again. Since 59 < 60, Li Si is not a valid result. At this point, all data has been read, and the result Zhang San is returned.
[0069] In the example above, the age fields 21 and 19 were read once, but not used in the calculation, wasting memory and increasing computation time. In real-world applications, when the data volume is much larger, often reaching tens of millions of rows and involving dozens of fields, such a large amount of data can lead to significant resource waste.
[0070] Based on this, the embodiments of this application perform row and column transformation in step 100, filter attribute information rows in step 200, and filter returned objects in step 300 below. This not only enables SQL request processing to be executed in the DPU, but also improves the efficiency of SQL request processing and effectively saves storage space.
[0071] Step 300: Based on the attribute information row corresponding to the target after field filtering, perform field filtering on the attribute information row corresponding to the returned object to obtain the request result data corresponding to the SQL request statement.
[0072] As can be seen from the above description, the SQL request processing method provided in this application embodiment can reduce the computation and storage of redundant data brought to the CPU by processing SQL request statements, thereby reducing the CPU resource utilization rate and improving the reliability of the CPU in SQL retrieval, and effectively improving the processing efficiency of SQL request statements.
[0073] To further improve the efficiency and reliability of row-to-column transformation of data tables in a row-oriented database, an SQL request processing method is provided in an embodiment of this application, see [link to relevant documentation]. Figure 2 or Figure 3 The SQL request processing method, prior to step 100, specifically includes the following:
[0074] Step 010: Based on the data reading operator preset in the DPU, read the data table from the row-based database outside the DPU. Each column in the data table is used to store all fields corresponding to each attribute type, and each column in the data table corresponds one-to-one with each data type.
[0075] In one or more embodiments of this application, the data reading operator can be a scanning operator, which scans data and reads it into the DPU. This operator, through the KPU architecture, can perform asynchronous computation, simultaneously reading data while performing row and column transformations and partial computation.
[0076] In other words, in the SQL request processing method provided in this application, before executing the SQL statement, the data is read and stored in the form of columns by the data read operator on the DPU, so as to improve the efficiency and reliability of converting data tables in row-based databases into rows and columns.
[0077] To further improve the reliability of row-to-column transformation of data tables in a row-oriented database and to enhance the efficiency of SQL request processing, an SQL request processing method is provided in this application embodiment, see [link to relevant documentation]. Figure 2 or Figure 3 Step 100 in the SQL request processing method specifically includes the following:
[0078] Step 110: Call the row-column transformation operator preset in the DPU to perform row-column transformation on the data table to obtain the corresponding columnar storage data table. Each row in the columnar storage data table is a different attribute information row, and each attribute information row is used to store all fields corresponding to each attribute type. Each attribute information row in the columnar storage data table corresponds one-to-one with each data type.
[0079] It is understood that each column in the columnar storage data table represents an independent record, and each record contains a field corresponding to each attribute type. Each row is used to store all fields corresponding to each attribute type, and each row in the columnar storage data table corresponds one-to-one with each of the data types.
[0080] In one example of this application, the columns of the columnar storage data table refer to the rows other than the first column, which is used only to represent the various data types.
[0081] Based on the aforementioned Table 1, in one example, after transforming Table 1 into rows and columns, a corresponding columnar storage data table is obtained as shown in Table 2. Each row in the columnar storage data table is used to store all fields corresponding to the three attribute types: "name", "age" and "score".
[0082] Table 2 Columnar Storage Data Table
[0083] Name Zhang San Li Si age 21 19 Fraction 95 59
[0084] In one or more embodiments of this application, the row-column transformation operator can be a scanning operator, which scans data and reads it into the DPU. This operator, through the KPU architecture, can perform asynchronous computation, simultaneously reading data and performing row-column transformations and partial computations.
[0085] To further improve the efficiency of SQL request processing, in an embodiment of this application, a method for processing SQL requests is provided, see [link to relevant documentation]. Figure 3 The SQL request processing method further includes the following content between steps 110 and 200:
[0086] Step 020: Receive the SQL request statement, and read the specified data table, determine the attribute type of the target, the judgment condition, and the attribute type of the returned object from the SQL request statement.
[0087] Specifically, by first performing row and column transformation on all relevant data tables in step 100 and then storing them locally in the DPU, and then in step 020, each time an SQL request statement is received, it is not necessary to retrieve the database again. Instead, the columnar storage data table that uniquely corresponds to the data table specified by the SQL request statement that has been row and column transformed and stored locally in the DPU can be used to execute step 200, thereby further improving the efficiency of SQL request processing.
[0088] To further improve the efficiency and effectiveness of filtering attribute information rows corresponding to the target, an SQL request processing method is provided in this application embodiment, see [link to relevant documentation]. Figure 2 or Figure 3Step 200 in the SQL request processing method specifically includes the following:
[0089] Step 210: Invoke the binary operation operator preset in the DPU, scan the columnar storage data table according to the attribute types corresponding to the judgment target and the return object specified in the SQL request statement, so as to obtain the attribute information row corresponding to the judgment target and the attribute information row corresponding to the return object, and perform field filtering on the attribute information row corresponding to the judgment target based on the judgment conditions specified in the SQL request statement.
[0090] Taking Table 2 as an example, the DPU reads the attribute information row "95,59" corresponding to the attribute type "score" in Table 2, and then filters the attribute information row corresponding to the attribute type "score" according to the judgment condition "score>60". The field "95" is the field that meets the judgment condition, and "59" is the field that does not meet the judgment condition.
[0091] In one or more embodiments of this application, the binary operation operator can be a scan operator, which scans data and reads it into the DPU. This operator, through the KPU architecture, can perform asynchronous computation, simultaneously reading data, performing row and column transformations, and conducting partial computations.
[0092] To further improve the efficiency and effectiveness of filtering attribute information fields corresponding to the returned object, in an SQL request processing method provided in this application embodiment, see [link to relevant documentation]. Figure 3 The SQL request processing method further includes the following content between steps 200 and 300:
[0093] Step 030: Invoke the binary operation operator, in the attribute information row corresponding to the judgment target, mask the fields that meet the judgment conditions with a first identifier, and mask the fields that do not meet the judgment conditions with a second identifier, wherein the first identifier and the second identifier are different.
[0094] Taking Table 2 as an example, since the DPU only reads the attribute information row "95,59" corresponding to the attribute type "Score" and cannot return the result for the "Name" field, the result can be saved as a mask. This mask can be 1, 0, where 1 is the first identifier and 0 is the second identifier. 1 represents a field that meets the judgment condition, and 0 represents a field that does not meet the judgment condition.
[0095] In other words, in this embodiment of the application, when executing SQL statements, data is calculated on a row-by-row basis, and a mask is generated after the calculation.
[0096] To further improve the efficiency and effectiveness of filtering attribute information fields corresponding to the returned object, in an SQL request processing method provided in this application embodiment, see [link to relevant documentation]. Figure 2 or Figure 3 Step 300 in the SQL request processing method specifically includes the following:
[0097] Step 310: Call the logical summation operator preset in the DPU, and perform a logical summation operation on the first identifier in the attribute information row corresponding to the judgment target and each field in the attribute information row corresponding to the returned object in the columnar storage data table to obtain the filtered attribute information row corresponding to the returned object, and use the filtered attribute information row corresponding to the returned object as the request result data corresponding to the SQL request statement.
[0098] Taking Table 2 as an example, a logical summation (AND) operation is performed between the row containing the first identifier of the field that meets the judgment condition and the attribute information row corresponding to the returned object. The mask corresponding to "Zhang San" is 1, indicating that the field "Zhang San" meets the judgment condition and is returned; the mask corresponding to "Li Si" is 0, indicating that "Li Si" does not meet the judgment condition and is not returned.
[0099] Step 320: Output the request result data corresponding to the SQL request statement.
[0100] In other words, this embodiment uses a mask and a target result row to perform a logical summation operation to filter out the results that meet the conditions. Taking Table 2 as an example, the DPU in this embodiment does not need to read the data in the age row at all. When the amount of data is large, this method can effectively save storage space and improve the efficiency of the operation.
[0101] In one or more embodiments of this application, the logical summation operator can be a scan operator (i.e., a SCAN operator), and the function of the scan operator is to scan data and read it into the DPU. This operator can perform asynchronous computation through the KPU architecture, and can perform row and column transformations and partial computations simultaneously while reading data.
[0102] It should also be noted that the SQL request processing methods provided in this application embodiment are all performed by DPU. By calling the row-column conversion operator on the DPU, the data is quickly converted from row mode to column mode. Then, the binary operation operator on the DPU scans only the key condition rows for judgment and generates a mask. Then, the SCAN operator on the DPU performs a logical summation operation on the mask and the potential result rows, selects the correct result and returns it.
[0103] Based on this, the embodiments of this application use the DPU to perform column operations on the data instead of the CPU's row operations, thereby improving computing efficiency and reducing the storage and computation of redundant data.
[0104] This application also provides an apparatus for executing all or part of the SQL request processing method described above, the apparatus being disposed in a data processor (DPU) based on a kernel processor (KPU) architecture. See [link to relevant documentation]. Figure 4 The SQL request processing device specifically includes the following components:
[0105] The row-column conversion module 10 is used to convert the data table in the row-based database into rows and columns to obtain the attribute information rows corresponding to each attribute type. Each attribute information row contains all the fields of an attribute type in the data table.
[0106] The target filtering module 20 is used to select the attribute information row corresponding to the target and the attribute information row corresponding to the returned object from each attribute information row according to the attribute types of the target and the returned object specified in the SQL request statement, and to perform field filtering on the attribute information row corresponding to the target based on the judgment conditions specified in the SQL request statement.
[0107] The object filtering module 30 is used to filter the attribute information row corresponding to the returned object based on the attribute information row corresponding to the judgment target after field filtering, so as to obtain the request result data corresponding to the SQL request statement.
[0108] The embodiments of the SQL request processing apparatus provided in this application can be used to execute the processing flow of the embodiments of the SQL request processing method described above. Its functions will not be repeated here, but can be referred to the detailed description of the embodiments of the SQL request processing method described above.
[0109] As can be seen from the above description, the SQL request processing device provided in this application embodiment can reduce the computation and storage of redundant data brought to the CPU by processing SQL request statements, thereby reducing the CPU resource utilization rate and improving the reliability of the CPU in SQL retrieval, and effectively improving the processing efficiency of SQL request statements.
[0110] To further improve the efficiency and reliability of row-to-column transformation of data tables in a row-oriented database, an SQL request processing device is provided in an embodiment of this application, see [link to relevant documentation]. Figure 5 The SQL request processing device also specifically includes the following:
[0111] The data reading module 01 is used to read a data table from a row-based database outside the DPU based on a preset data reading operator in the DPU before performing row-column conversion on the data table in the row-based database. Each column in the data table is used to store all fields corresponding to each attribute type, and each column in the data table corresponds one-to-one with each data type.
[0112] This application also provides a DPU based on a KPU architecture, wherein the DPU includes an SQL request processing device, which is used for the SQL request processing method described in the foregoing embodiments; see also Figure 6 The SQL request processing device communicates with a row-oriented database external to the DPU; and it also communicates with an interactive query module in the distributed computing platform to receive SQL request statements from that module. The distributed computing platform can be Spark, and the interactive query module can be Spark SQL.
[0113] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned SQL request processing method. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.
[0114] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.
[0115] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0116] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.
[0117] The above description is merely a preferred embodiment of this application and is not intended to limit this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A method for processing SQL requests, characterized in that, The method is executed by a data processor (DPU) based on a kernel processor (KPU) architecture, and the method includes: The data tables in the row-based database are transformed into rows and columns to obtain attribute information rows corresponding to each attribute type. Each attribute information row contains all the fields of an attribute type in the data table. Based on the attribute types corresponding to the judgment target and the returned object specified in the SQL request statement, the attribute information row corresponding to the judgment target and the attribute information row corresponding to the returned object are selected from each of the attribute information rows respectively, and the attribute information row corresponding to the judgment target is filtered by field based on the judgment conditions specified in the SQL request statement; The binary operation operator preset in the DPU is invoked. In the attribute information row corresponding to the judgment target, the fields that meet the judgment conditions are masked with a first identifier, and the fields that do not meet the judgment conditions are masked with a second identifier, wherein the first identifier and the second identifier are different. The logical summation operator preset in the DPU is invoked. In the columnar storage data table, the first identifier in the attribute information row corresponding to the judgment target is logically summed with each field in the attribute information row corresponding to the returned object to obtain the filtered attribute information row corresponding to the returned object. The filtered attribute information row corresponding to the returned object is used as the request result data corresponding to the SQL request statement. The request result data corresponding to the SQL request statement is then output.
2. The SQL request processing method according to claim 1, characterized in that, Before performing row-to-column transformation on the data tables in the row-based database, the following steps are also included: Based on the data reading operator preset in the DPU, a data table is read from a row-based database outside the DPU. Each column in the data table is used to store all fields corresponding to each attribute type, and each column in the data table corresponds one-to-one with each attribute type.
3. The SQL request processing method according to claim 1, characterized in that, The process of transforming the data tables in the row-based database into rows and columns to obtain the attribute information rows corresponding to each attribute type includes: The row and column transformation operator preset in the DPU is invoked to perform row and column transformation on the data table to obtain the corresponding columnar storage data table. Each row in the columnar storage data table is a different attribute information row, and each attribute information row is used to store all fields corresponding to each attribute type. Each attribute information row in the columnar storage data table corresponds one-to-one with each attribute type.
4. The SQL request processing method according to claim 1, characterized in that, After performing row and column transformation on the data tables in the row-based database, the following steps are also included: It receives SQL request statements, reads the specified data table from the SQL request statement, determines the attribute type of the target, the judgment conditions, and the attribute type of the returned object.
5. The SQL request processing method according to claim 3, characterized in that, The step involves selecting the attribute information rows corresponding to the judgment target and the return object from each of the attribute information rows according to the attribute types specified in the SQL request statement, and then filtering the attribute information rows corresponding to the judgment target based on the judgment conditions specified in the SQL request statement. This includes: The binary operation operator preset in the DPU is invoked to scan the columnar storage data table according to the attribute types corresponding to the judgment target and the return object specified in the SQL request statement, so as to obtain the attribute information row corresponding to the judgment target and the attribute information row corresponding to the return object, and to filter the attribute information row corresponding to the judgment target based on the judgment conditions specified in the SQL request statement.
6. An SQL request processing apparatus, characterized in that, The device is installed in a data processor (DPU) based on a kernel processor (KPU) architecture, and the device includes: The row-column transformation module is used to transform the data table in the row-based database into rows and columns to obtain the attribute information rows corresponding to each attribute type. Each attribute information row contains all the fields of an attribute type in the data table. The target filtering module is used to select the attribute information row corresponding to the target and the attribute information row corresponding to the returned object from each attribute information row according to the attribute types of the target and the returned object specified in the SQL request statement, and to perform field filtering on the attribute information row corresponding to the target based on the judgment conditions specified in the SQL request statement. The SQL request processing device is also used to execute: calling a binary operation operator preset in the DPU, in the attribute information row corresponding to the judgment target, masking the fields that meet the judgment conditions with a first identifier, and masking the fields that do not meet the judgment conditions with a second identifier, wherein the first identifier and the second identifier are different; The SQL request processing device further includes: The object filtering module is used to call the logical summation operator preset in the DPU, and perform a logical summation operation on the first identifier in the attribute information row corresponding to the judgment target and each field in the attribute information row corresponding to the returned object in the columnar storage data table to obtain the filtered attribute information row corresponding to the returned object, and use the filtered attribute information row corresponding to the returned object as the request result data corresponding to the SQL request statement; and output the request result data corresponding to the SQL request statement.
7. The SQL request processing apparatus according to claim 6, characterized in that, Also includes: The data reading module is used to read data tables from a row-based database outside the DPU based on a preset data reading operator in the DPU before performing row-column conversion on the data tables in the row-based database. Each column in the data table is used to store all fields corresponding to each attribute type, and each column in the data table corresponds one-to-one with each attribute type.
8. A DPU based on a KPU architecture, characterized in that, The DPU is equipped with an SQL request processing device, which is used to execute the SQL request processing method according to any one of claims 1 to 5; The SQL request processing device communicates with a row-based database outside the DPU; and the SQL request processing device communicates with an interactive query module in the distributed computing platform to receive SQL request statements from the interactive query module.