Data processing method, and device, storage medium and computer program product

Abstract

Provided in the present application are a data processing method, and a device, a storage medium and a computer program product. The method comprises: acquiring a query statement; generating an execution plan for the query statement, and adding target computing nodes between first computing nodes and second computing nodes of the execution plan, wherein the first computing nodes are computing nodes for executing target computation operations, the target computing nodes are parent nodes of the first computing nodes, and the second computing nodes are parent nodes of the target computing nodes; executing the execution plan, so as to provide, at the first computing nodes, acquired tuples to the target computing nodes; and storing, in the target computing nodes, the tuples provided by the first computing nodes, performing batch processing on a plurality of stored tuples according to the target computation operations, and respectively providing a plurality of obtained output results to the second computing nodes.

Description

Technical Field of Data Processing Method, Device, Storage Medium and Computer Program Product

[0001] Embodiments of the present application relate to the field of data processing, and in particular, to a data processing method, device, storage medium and computer program product. Background Art

[0002] In a database system, an execution plan is a data structure generated according to a query statement, which describes the operation process taken by the database system to execute the query. It is composed of multiple computing nodes. Taking a database system based on the volcano model as an example, different computing nodes will be logically organized into a tree structure, and each computing node is responsible for performing different processing operations. Each computing node receives the output of its child computing node as its own input, and then passes its output upward to the computing node of its parent node. In the database system, this data transfer is realized through tuples. A tuple is also a record in a database table. A computing node receives one tuple as input at a time, performs calculations within the computing node, and generates one tuple as output.

[0003] As can be seen from the above description, the computing nodes perform calculations one by one. As the computing tasks of modern database systems become more and more complex and diversified, such as in various database scenarios that support machine learning, confidential computing, and heterogeneous computing involving a large number of tuple calculations with high performance, if the one-by-one calculation method is still used, the calculation process will be cumbersome, and at the same time, it may bring more computing resource overhead and poor calculation effects. Summary of the Invention

[0004] Embodiments of the present application provide a data processing method, device, storage medium and computer program product to solve the problems that the calculation process is cumbersome and may bring more computing resource overhead when using the one-by-one calculation method to execute the calculation tasks of the database system in the prior art.

[0005] In a first aspect, a data processing method is provided in an embodiment of the present application, including: obtaining a query statement; generating an execution plan for the query statement, and adding a target computing node between a first computing node and a second computing node in the execution plan; where the first computing node is a computing node that executes a target operation; the target computing node is used as the parent node of the first computing node; the second computing node is used as the parent node of the target computing node; executing the execution plan so that the first computing node provides the obtained tuple as an output result to the target computing node; The target computing node stores the tuples provided by the first computing node, performs batch processing on the stored multiple tuples according to the target computing operation, and provides multiple output results obtained by the batch processing to the second computing node respectively.

[0006] In a second aspect, an embodiment of the present application provides a computing device, comprising a storage component and a processing component; the storage component stores one or more computer program instructions, the computer program instructions are called and executed by the processing component, and the processing component executes the one or more computer program instructions to implement the data processing method as described in the first aspect.

[0007] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a computer to implement the data processing method as described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer program product, which stores a computer program, and when the computer program is executed by a computer, implements the data processing method as described in the first aspect.

[0009] In an embodiment of the present application, a query statement can be obtained, an execution plan for the query statement can be generated, and a target computing node can be added between a first computing node and a second computing node in the execution plan, wherein the first computing node is a computing node that performs a target computing operation, the target computing node is used as a parent node of the first computing node, and the second computing node is used as a parent node of the target computing node, and then the execution plan is executed to provide the obtained tuple as an output result to the target computing node at the first computing node, save the tuple provided by the first computing node at the target computing node, and batch process the saved multiple tuples according to the target computing operation, and provide the multiple output results obtained by the batch processing to the second computing node respectively. By adding the target computing node as the parent node of the computing node that executes the target computing operation in the execution plan, the tuple obtained by the first computing node can be output to the target computing node for storage, and multiple tuples can be batch processed at the target computing node, and multiple output results are provided to the second computing node, that is, the parent node of the target computing node. Compared with the traditional solution in which the computing node accepts a tuple as input for calculation and generates a tuple as output, the one-by-one calculation method realizes batch execution, simplifies the calculation process, and can better handle computing tasks in scenarios involving a large number of tuple calculations, thereby expanding the applicable scenarios of computing task execution.

[0010] These aspects or other aspects of the present application will be more clearly understood in the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following briefly introduces the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0012] FIG. 1 shows a flowchart of an embodiment of a data processing method provided by the present application;

[0013] FIG. 2 shows a schematic diagram of the conversion of a custom type in an actual application of an embodiment of the present application;

[0014] FIG. 3 shows a schematic structural diagram of an execution plan in an actual application of an embodiment of the present application;

[0015] FIG. 4 shows a schematic diagram of scenario interaction in an actual application of an embodiment of the present application;

[0016] FIG. 5 shows a schematic structural diagram of an embodiment of a data processing apparatus provided by the present application;

[0017] FIG. 6 shows a schematic structural diagram of an embodiment of a computing device provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] In order to enable those skilled in the art to better understand the solutions of the present application, the following clearly and completely describes the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application.

[0019] In some processes described in the specification, claims and the above-mentioned drawings of the present application, there are multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The operation numbers such as 101, 102, etc. are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are different types.

[0020] In a database system, an execution plan is a data structure generated based on a query statement, which describes the operation process that the database system takes to execute the query. It consists of multiple computing nodes composed of operators. Taking a database system based on the volcano model as an example, different computing nodes will be logically organized into a tree structure, and each computing node is responsible for performing different processing operations. Each computing node accepts the output of its child computing nodes as its own input, and then passes its own output upward to the computing nodes of its parent node. In the database system, this data transfer is achieved through tuples, which also refer to a record in a database table. A computing node accepts one tuple as input at a time, performs calculations within the computing node, and generates one tuple as output.

[0021] As can be seen from the above description, the computing nodes perform calculations one by one. However, with the increasing complexity and diversification of the computing tasks in modern database systems, such as in various database scenarios that support machine learning, confidential computing, and heterogeneous computing with high performance and involve a large number of tuple calculations, if the one-by-one calculation method is still used, it will lead to a cumbersome calculation process, and at the same time may bring more computing resource overhead and poor calculation results.

[0022] To solve the above technical problems, the inventors proposed the technical solution of this application. In the embodiments of this application, a query statement is obtained; an execution plan for the query statement is generated, and a target computing node is added between the first computing node and the second computing node in the execution plan; wherein, the first computing node is a computing node that performs a target operation; the target computing node is used as the parent node of the first computing node; the second computing node is used as the parent node of the target computing node; the execution plan is executed to provide the tuples obtained at the first computing node as output results to the target computing node; the tuples provided by the first computing node are saved at the target computing node, and the saved multiple tuples are batch-processed according to the target operation, and the multiple output results obtained from the batch processing are respectively provided to the second computing node. The target computing node is used as the parent node of the first computing node; the second computing node is used as the parent node of the target computing node; execute the execution plan to provide the tuples obtained by the first computing node as output results to the target computing node; save the tuples provided by the first computing node at the target computing node, batch-process the saved multiple tuples according to the target operation, and respectively provide the multiple output results obtained from the batch processing to the second computing node.

[0023] In the embodiment of the present application, by adding the target computing node as the parent node of the computing node that executes the target computing operation in the execution plan, the tuple obtained by the first computing node can be output to the target computing node for storage, and multiple tuples are batch processed at the target computing node, and multiple output results are provided to the second computing node, that is, the parent node of the target computing node. Compared with the one-by-one calculation method in which the computing node accepts a tuple as input for calculation and generates a tuple as output in the traditional solution, batch execution is achieved, the calculation process is simplified, and the computing tasks in the scenario involving the calculation of a large number of tuples can be better processed, thereby expanding the applicable scenarios of the execution of computing tasks.

[0024] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0025] FIG1 shows a flow chart of an embodiment of a data processing method provided by the present application. The technical solution of the present embodiment can be executed by a database engine. In practical applications, the technical solution of the embodiment of the present application can be applied to a database processing system composed of a user end and a server end. The database engine can be deployed in the server end. The method may include the following steps 101 to 104.

[0026] 101: Get the query statement.

[0027] In this embodiment, the data processing method can be applied to the server. Optionally, the query statement can be sent by the user end, and the server end can receive the query statement sent by the user end.

[0028] 102: Generate an execution plan for the query statement, and add a target computing node between the first computing node and the second computing node in the execution plan.

[0029] The first computing node is a computing node that performs a target computing operation, and the second computing node is the original parent node of the first computing node. After adding the target computing node, the target computing node is used as the parent node of the first computing node, and the second computing node is used as the parent node of the target computing node.

[0030] The execution plan can be implemented as various data structures such as a tree structure, a graph structure, etc. Taking a database system based on the volcano model as an example, an execution plan tree for a query statement can be generated. The execution plan tree can include multiple computing nodes. For the sake of convenience in description, the computing node that executes the target operation can be called the first computing node, and the original parent node of this first computing node can be called the second computing node. Among them, the target operation can include, for example, numerical operation, comparison operation, logical operation, and so on. node's original parent node is called the second computing node. Among them, the target operation can include, for example, numerical operation, comparison operation, logical operation, and so on.

[0031] Furthermore, a target computing node can be added between the first computing node and the second computing node. At this time, the added target computing node can be used as the parent node of the first computing node, and the second computing node can be used as the parent node of this target computing node.

[0032] Optionally, there can be multiple first computing nodes in the above-mentioned execution plan. For each first computing node, a corresponding target computing node can be added above this first computing node as the parent node of this first computing node.

[0033] 103: Execute this execution plan to provide the obtained tuples as output results to the target computing node at the first computing node.

[0034] Execute the above-mentioned execution plan, use the first computing node to obtain tuples, and provide the obtained tuples as output results to the target computing node. That is to say, different from the traditional solution where the first computing node needs to perform the target operation on the data in the tuples, in the solution of the embodiment of the present application, the target operation is no longer actually performed in the first computing node, and all the obtained tuples will pass the verification and be provided to its upper-layer node, that is, only the obtained tuples are provided to its upper-layer node.

[0035] 104: Save the tuples provided by the first computing node at the target computing node, batch process the saved multiple tuples according to the target operation, and provide the multiple output results obtained from the batch processing to the second computing node respectively.

[0036] The target computing node can first save the tuples provided by the first computing node.

[0037] And batch process the saved multiple tuples according to the target operation, and provide the obtained multiple output results to the second computing node.

[0038] Optionally, when the tuple storage capacity of the target computing node meets the processing requirements, the target computing node may process multiple stored tuples according to the target operation. Herein, the processing requirements may be set according to the actual application scenario. For example, it may be set that when the tuple storage capacity meets a preset quantity, multiple stored tuples are batch-processed, or when the tuple storage capacity reaches the storage threshold, multiple stored tuples are batch-processed, or when it is determined that the first computing node has provided all the obtained tuples and no longer provides tuples, multiple stored tuples are batch-processed, and so on.

[0039] In an actual application, the technical solution of the embodiment of the present application can be applied to scenarios that require invoking external hardware to perform target operations. For example, in computing scenarios such as encryption processing or acceleration processing, the external hardware to be invoked may be, for example, invoking a graphics processing unit (GPU) for operation, invoking an external field programmable gate array (FPGA) for operation, etc. Therefore, optionally, in the target computing node, external hardware may be invoked to batch-process data objects in multiple stored tuples according to the target operation.

[0040] Specifically, the process of the target computing node batch-processing multiple stored tuples according to the target operation and providing the obtained multiple output results to the second computing node will be described in subsequent embodiments.

[0041] According to the above output results, the query result corresponding to the execution plan can be obtained. Optionally, the server may also feedback the query result corresponding to the execution plan to the user side.

[0042] In this embodiment, a query statement can be obtained, an execution plan for the query statement can be generated, and a target computing node can be added between a first computing node and a second computing node in the execution plan, wherein the first computing node is a computing node that performs a target computing operation, the target computing node is used as a parent node of the first computing node, and the second computing node is used as a parent node of the target computing node. The execution plan is then executed to provide the obtained tuple as an output result to the target computing node at the first computing node, save the tuple provided by the first computing node at the target computing node, perform batch processing on the saved multiple tuples according to the target computing operation, and provide the multiple output results obtained by the batch processing to the second computing node respectively. By adding the target computing node as the parent node of the computing node that performs the target computing operation in the execution plan, the tuple obtained by the first computing node can be powdered out to the target computing node for storage, and multiple tuples can be batch processed at the target computing node, and multiple output results are provided to the second computing node, that is, the parent node of the target computing node. Compared with the traditional solution in which the computing node accepts a tuple as input for calculation and generates a tuple as output, the one-by-one calculation method realizes batch execution, simplifies the calculation process, and can better handle computing tasks in scenarios involving a large number of tuple calculations, expanding the applicable scenarios of computing task execution.

[0043] In practical applications, the execution plan may include multiple computing nodes. During the execution of the computing task, not every computing node will perform the target computing operation. Therefore, the computing node that performs the target computing operation, that is, the first computing node, can be determined, and then the target computing node is added thereto. Therefore, in some embodiments, the above-mentioned generation of the execution plan of the query statement and adding the target computing node between the first computing node and the second computing node of the execution plan may include: generating the execution plan of the query statement, and when the query statement involves the target computing operation that meets the preset conditions, adding the target computing node between the first computing node and the second computing node of the execution plan.

[0044] Specifically, the preset condition may include one or more of the following implementations: operation on a specified data type; operation that meets a predetermined operation requirement; operation that calls external hardware; and operation that generates a custom type data according to a pre-configured custom type and the custom type's operation requirement.

[0045] Among them, the specified data types can include, for example, numeric types, string types, etc. Taking the specified data type as a numeric type as an example, at least one computing node for performing arithmetic operations on the numeric type can be determined from multiple computing nodes of the execution plan as the first computing node, and a corresponding target computing node can be added thereto as the parent node.

[0046] The predetermined arithmetic requirements can include, for example, comparison operations, logical operations, etc. Taking the predetermined arithmetic requirement as a comparison operation as an example, at least one computing node for performing the comparison operation can be determined from multiple computing nodes of the execution plan as the first computing node, and a corresponding target computing node can be added thereto as the parent node.

[0047] The arithmetic operations of calling external hardware can include, for example, calling a GPU for arithmetic, calling an FPGA for arithmetic, etc. At least one computing node for calling external hardware can be determined from multiple computing nodes of the execution plan as the first computing node, and a corresponding target computing node can be added thereto as the parent node.

[0048] The custom type can be different from primitive data types such as numeric types, string types, etc., and can be configured according to the actual application scenario. The arithmetic requirements of the custom type can include generating custom type data for arithmetic operations on primitive data types and generating custom type data for arithmetic operations on custom types. Thus, at least one computing node for performing the arithmetic operation of generating custom type data according to the preconfigured custom type and the arithmetic requirements of the custom type can be determined from multiple computing nodes of the execution plan as the first computing node, and a corresponding target computing node can be added thereto as the parent node.

[0049] By setting the above preset conditions, the determination of the first computing node in the execution plan can be achieved, which is convenient for adding a target computing node above the first computing node, and the target computing node performs batch arithmetic on multiple tuples provided by the corresponding first computing node to meet the batch processing requirements.

[0050] The following takes the target arithmetic operation as an example of the arithmetic operation of generating custom type data according to the preconfigured custom type and the arithmetic requirements of the custom type to illustrate this data processing method.

[0051] Each primitive data type can define its own corresponding custom type. For example, the primitive numeric type can correspond to a custom numeric type, the primitive string type can correspond to a custom string type, the primitive boolean type can correspond to a custom boolean type, and so on.

[0052] The operation requirements of a custom type can include generating custom type data for operation operations on primitive data types and generating custom type data for operation operations on custom types. For example, it is possible to pre-configure generating custom numeric type data for numeric operation operations on primitive numeric types, generating custom boolean type data for comparison operations on custom boolean types, and so on.

[0053] In some embodiments, in the case of an operation operation involving the above target operation operation, the operation result of the first computing node can be set to pass the verification. For example, the target operation operation includes generating custom boolean type data for a comparison operation on a primitive data type, and also generating custom boolean type data for a comparison operation on custom boolean type data. The operation result of the first computing node can be set to "true", that is, for any comparison operation on primitive data type data and any comparison operation on custom data type data, the obtained comparison result is "true". Thus, the first computing node can output all the data participating in the target operation operation and provide it to the target computing node.

[0054] Moreover, in the target computing node, the data participating in the target operation operation in the saved tuple can be batch-processed according to the target operation operation and generate multiple operation results of the custom type. For example, the target operation operation includes generating custom boolean type data for a comparison operation on a primitive data type, and also generating custom boolean type data for a comparison operation on custom boolean type data. In the target computing node, a comparison operation can be performed on the data participating in the target operation operation to generate an operation result of the custom boolean type.

[0055] And, considering that in actual applications, the output result provided by the target computing node to the second computing node is usually primitive type data. Therefore, in the target computing node, the operation result of the custom type can also be converted into the corresponding primitive data type.

[0056] Therefore, the method for the first computing node to provide the obtained tuple as an output result to the target computing node can include:

[0057] In the first computing node, when performing the target operation operation on the obtained tuple, determining that the operation result passes the verification, saving the data participating in the target operation operation as a data object in the tuple, and providing the tuple as the output result to the target computing node.

[0058] Further, the method for batch processing the saved multiple tuples according to the target operation can include:

[0059] In the target computing node, batch process the data objects in the saved multiple tuples according to the target operation to generate multiple operation results of a custom type;

[0060] Convert the multiple operation results into corresponding original data types respectively;

[0061] Determine multiple output results to be provided to the second computing node according to the multiple operation results.

[0062] For example, after the multiple operation results generated by the target computing node are converted into corresponding original data types, including multiple "true" operation results and multiple "false" operation results, multiple output results that meet the query request to be provided to the second computing node can be determined. For example, determine the tuples corresponding to the multiple "true" operation results as the multiple output results to be provided to the second computing node.

[0063] In this embodiment, when the target operation is performed on the obtained tuples in the first computing node, it is determined that the verification of the operation results passes, so that the first computing node no longer actually performs the target operation, but saves the data participating in the target operation to the tuples and provides them to the target computing node for batch processing, realizing the batch processing of tuples.

[0064] To achieve batch processing, a target computing node is added between the first computing node and the second computing node. The first computing node directly provides the obtained tuples as output results to the target computing node, and the target computing node performs batch processing on multiple tuples according to the target operation. It can be understood that in this process, the target operation on the data in the tuples is delayed and postponed to the target computing node for batch execution, while the first computing node no longer performs the target operation. To avoid modifying the structure of the first computing node in the execution plan, the modification method of the query statement can be adopted. Therefore, in some embodiments, the method of generating an execution plan for a query statement and adding a target computing node between the first computing node and the second computing node in the execution plan may include: detecting that the query statement involves a target operation; rewriting the query statement to enclose the operation expression corresponding to the target operation with a target execution function outside; parsing the query statement to generate an execution plan, and when the query statement includes the target execution function, adding a target computing node between the first computing node and the second computing node in the execution plan.

[0065] Specifically, when detecting that the query statement involves a target operation, such as when the query statement includes the operation expression corresponding to the target operation, the query statement can be rewritten to enclose the operation expression corresponding to the target operation with a target execution function outside.

[0066] Taking the query statement "SELECT * FROM t WHERE a > encrypt(3)" as an example, this query statement can be implemented for a fully encrypted database (a database solution that uses ciphertext processing technology to achieve data security protection, which can rely on encryption technologies such as Trusted Execution Environment (TEE) and Fully Homomorphic Encryption (FHE)). This query statement can represent retrieving all tuples from table t where the data content in column a is greater than the encrypted data "encrypt(3)", that is, comparing the data in column a of the tuples in table t with the encrypted data "encrypt(3)". The target operation involved is a comparison operation, and the operation expression corresponding to this target operation is a > encrypt(3). Then the target execution function can be represented as the force function, for example. This target execution function can be called by the computing node in the execution plan to delay the target operation on the data in the tuples from the first computing node to the target computing node for execution, thereby achieving batch processing.

[0067] After that, the rewritten query statement can be parsed to generate an execution plan. The rewritten query statement can be implemented as SELECT * FROM t WHERE force(a > encrypt(3)). Also, when the target execution function is included in the query statement, a target calculation node is added between the first calculation node and the second calculation node in the execution plan.

[0068] By rewriting the query statement by wrapping the operation expression corresponding to the target operation in the query statement with the target execution function, generating an execution plan using the rewritten query statement, and adding a target calculation node above the first calculation node, it is possible to directly provide the obtained tuples as output results to the target calculation node without modifying the structure of the first calculation node, that is, delaying the target operation on the data in the tuples to the target calculation node for batch processing. Execute, thereby realizing batch processing.

[0069] On this basis, still taking the target operation as an example of generating custom type data operations according to the pre-configured custom type and the operation requirements of the custom type, this data processing method will be described.

[0070] In some embodiments, the above target execution function can be used to set the operation result of the first calculation node to verified when the target operation is involved. For example, if the target operation includes generating custom boolean type data for comparison operations on the original data type and also generating custom boolean type data for comparison operations on the custom boolean type data, the operation result of the first calculation node can be set to "true". That is to say, for any comparison operation on the original data type data and any comparison operation on the custom data type data, the obtained comparison result is "true", so that the first calculation node can output all the data participating in the target operation and provide it to the target calculation node.

[0071] Also, the target execution function can also be used to batch process the data participating in the target operation in the saved tuples according to the target operation in the target calculation node and generate multiple operation results of the custom type. For example, if the target operation includes generating custom boolean type data for comparison operations on the original data type and also generating custom boolean type data for comparison operations on the custom boolean type data, in the target calculation node, comparison operations can be performed on the data participating in the target operation to generate operation results of the custom boolean type.

[0072] Moreover, considering that in practical applications, the output result provided by the target computing node to the second computing node is usually of the original type data, therefore, the target execution function can also be used to convert the operation result of the custom type into the corresponding original data type at the target computing node.

[0073] Therefore, the method for the first computing node to provide the obtained tuple as the output result to the target computing node may include: in the first computing node, call the target execution function to determine that the operation result passes the verification when performing the target operation on the obtained tuple, and save the data participating in the target operation as the data object in the tuple, and provide the tuple as the output result to the target computing node.

[0074] Furthermore, the method for batch processing the saved multiple tuples according to the target operation may include: in the target computing node, call the target execution function to batch process the data objects in the saved multiple tuples according to the target operation to generate multiple operation results of the custom type; convert the multiple operation results into the corresponding original data types respectively; determine multiple output results to be provided to the second computing node according to the multiple operation results.

[0075] For example, after the multiple operation results generated by the target computing node are converted into the corresponding original data types, including multiple "true" operation results and multiple "false" operation results, then it can be determined the multiple output results that meet the query request to be provided to the second computing node, such as determining the tuples corresponding to the multiple "true" operation results as the multiple output results to be provided to the second computing node. The multiple output results that meet the query request to be provided to the second computing node, such as determining the tuples corresponding to the multiple "true" operation results as the multiple output results to be provided to the second computing node.

[0076] In some embodiments, after determining the multiple output results, the above method may further include: storing the multiple output results in the processing order for the second computing node to sequentially obtain the multiple output results.

[0077] Specifically, the method for storing the multiple output results in the processing order may include: putting the multiple output results into the storage queue O in sequence according to the order in which the corresponding tuples are provided to the target computing node.

[0078] By storing the output results in the form of a queue to realize the sequential acquisition and spitting out of tuples by the target computing node, it can ensure the calculation correctness without modifying the original execution logic of the database.

[0079] Optionally, a ring buffer may be used to simultaneously ensure the relative order between output results and reduce data movement. Alternatively, a general data structure such as a linked list may be used to maintain the queue, which is not limited in this application.

[0080] The technical solution of the embodiment of the present application can be applied to a computing scenario for a fully secret database in a practical application. A fully secret database is a database solution that uses ciphertext processing technology to achieve data security protection, and can be implemented by relying on encryption technologies such as a trusted execution environment (Trusted Execution Environment, TEE for short), and fully homomorphic encryption (FHE for short). It is necessary to call external hardware to perform data processing in a trusted execution environment. In a fully secret database system, the data content exists in the form of ciphertext, and the ciphertext processing occurs and is calculated immediately, and the ciphertext is transmitted in the database as an intermediate form. Taking a fully secret database based on TEE as an example, the tuple contains secret data. In the traditional way, when performing calculations, it is usually firstly used to implement a secure call to the computing instance running in the TEE using untrusted conventional software, and then the secret data is decrypted in the computing instance, and the calculation is performed in plain text, and finally the calculation result is output in the form of ciphertext. This way of implementing calculations one by one requires secure calls, encryption and decryption of the computing instance in the TEE for each calculation, and the TEE is provided by external hardware. Calling external hardware is likely to bring about a large computing resource overhead. Therefore, the batch processing calculation solution proposed in this application will significantly optimize the above-mentioned fully encrypted database system.

[0081] In a fully secret database, since the data content exists in the form of ciphertext, it is usually necessary to define the ciphertext type and the operation type corresponding to the ciphertext type. Taking the numeric type as an example, in a plaintext database, a plaintext numeric type can be defined, such as the int4 type, which can represent a 4-byte plaintext numeric type. Correspondingly, in a fully secret database, a ciphertext numeric type can be defined, such as the enc_int4 type, which can represent an encrypted 4-byte numeric type.

[0082] When processing plaintext, digital arithmetic operations can include operations such as addition, subtraction, multiplication, and division, and comparison operations can include greater than, less than, and equal operations. Digital arithmetic operations on plaintext digital type data will generate digital type data, while comparison operations will generate Boolean type data. Correspondingly, when processing ciphertext, digital arithmetic operations on ciphertext digital type data will also generate ciphertext digital type data, and comparison operations can generate plaintext Boolean type data or ciphertext Boolean type data, which can be set according to the actual application scenario.

[0083] Corresponding to the above original digital types, custom types can include custom digital types, such as those that can be represented as delay_int4, delay_enc_mt4, etc., and custom Boolean types, such as those that can be represented as delay_bool.

[0084] The operation requirements for custom types can include generating custom digital type data for digital arithmetic operations on the original digital types, generating custom Boolean type data for comparison operations on the original digital types, generating custom digital type data for digital arithmetic operations on custom digital types, and generating custom Boolean type data for comparison operations on custom Boolean types, etc. For an example, refer to the schematic diagram showing an embodiment of a custom type and its operations in Figure 2.

[0085] Taking the data participating in the target operation as including digital type data, and the target operation as including digital operation and / or comparison operation for generating custom type data according to the pre-configured custom type and the operation requirements of the custom type as an example, in the target calculation node, the target execution function is called to batch process the data objects in the saved multiple tuples according to the target operation, and generating multiple operation results of the custom type may include: in the target calculation node, when the target operation is a digital operation for generating custom type data according to the pre-configured custom type and the operation requirements of the custom type, batch process the data objects in the saved multiple tuples according to the digital operation to generate an operation result of the custom digital type; when the target operation is a comparison operation for generating custom type data according to the pre-configured custom type and the operation requirements of the custom type, batch process the data objects in the saved multiple tuples according to the comparison operation to generate an operation result of the custom boolean type; when the target operation is a digital operation and a comparison operation for generating custom type data according to the pre-configured custom type and the operation requirements of the custom type, for the data objects in the saved multiple tuples, sequentially execute the digital operation and the comparison operation according to the operation order for batch processing. When the last operation is a digital operation, generate an operation result of the custom digital type, and when the last operation is a comparison operation, generate an operation result of the custom boolean type.

[0086] Further, converting the multiple operation results into the corresponding original data types may include: when the operation results include operation results of the custom digital type, converting the operation results of the custom digital type into operation results of the original digital type; and when the operation results include operation results of the custom boolean type, converting the operation results of the custom boolean type into operation results of the original boolean type.

[0087] Next, taking the data participating in the target operation as including ciphertext digital type data, and the target operation as including a comparison operation for generating custom type data according to the pre-configured custom type and the operation requirements of the custom type as an example, this data processing method will be described.

[0088] For example, still taking the query statement as SELECT * FROM t WHERE a > encrypt(3) as an example, this query statement table As shown, all tuples with data content in column a greater than encrypted 3 are retrieved from table t. Taking the common volcano model of relational databases as an example, the execution plan generated according to this query statement may include two computing nodes: a SCAN node and an Output node. Among them, the SCAN node can retrieve one tuple from the storage medium each time it executes, and complete the judgment on whether the data content in column a of the tuple is greater than encrypted 3. The Output node, as the parent node of the SCAN node, will continuously call the SCAN node to obtain the tuples output by the SCAN node until the SCAN node no longer outputs tuples.

[0089] In the embodiments of the present application, it is detected that the above query statement involves a target operation, and the operation expression corresponding to the target operation is a > encrypt(3). The computing node that executes the target operation is the SCAN node. Therefore, the SCAN node can be used as the first computing node, and the Output node can be used as the second computing node. Rewrite the query statement, wrap the force function around this operation expression, and the rewritten query statement is implemented as SELECT * FROM t WHERE force(a > encrypt(3)). Parse the rewritten query statement and generate an execution plan, and add a target computing node between the SCAN node and the Output node, which can be represented by a BUFFER node. For ease of understanding, FIG. 3 shows a schematic diagram of an embodiment of an execution plan.

[0090] As shown in FIG. 3, the SCAN node can retrieve one tuple from the storage medium each time it executes, and call the force function. When performing the judgment and verification of the WHERE statement, that is, when executing the target operation of a > encrypt(3) for the retrieved tuple, the operation result of each tuple is set to "true", that is, it is determined that the data content in column a of each tuple is greater than encrypted 3. All operation results are verified and passed, and the data participating in this target operation is saved as a data object in the tuple, and the tuple is provided as the output result to the BUFFER node.

[0091] The BUFFER node, as the parent node of the SCAN node, can save multiple tuples provided by the SCAN node. When the tuple storage capacity meets the processing requirements, it calls the force function to batch process the data objects in the saved multiple tuples according to the target operation of a > encrypt(3), generating multiple operation results of a custom Boolean type. Specifically, it can call the decryption component to decrypt the data objects in the tuples, process the decrypted data objects, and then encrypt the operation data obtained from the processing to obtain the operation results. Among them, the decryption component can be implemented as internal hardware or external hardware, and the present application does not limit this. As shown in Figure 3, the BUFFER node batch processes the data objects in the saved N tuples, and can generate N operation results of a custom Boolean type, where N is an integer greater than 1. After that, the BUFFER node can also convert the N operation results into the original Boolean type respectively.

[0092] After generating multiple operation results, the BUFFER node can also determine multiple output results to be provided to the Output node, that is, the second computing node, that is, determine the tuples with the operation result of "true" as the output results provided to the Output node. After that, the multiple output results can be sequentially placed in the storage queue in the order in which the corresponding tuples are provided to the BUFFER node.

[0093] The Output node, as the parent node of the BUFFER node, can call the BUFFER node to obtain tuples from the storage queue. Retrieve tuples.

[0094] In the embodiments of the present application, by adding a target computing node to the execution plan, the operation order of the traditional execution plan is changed. When the target computing node needs to return tuples to the second computing node, i.e., its parent node, it first continuously obtains the first computing node, i.e., its child node, saves the obtained multiple tuples, and then performs batch processing. Thus, the actual target operation on the data in the tuples is transferred from the first computing node to the target computing node for batch execution, realizing local caching and batch computing of tuples, and can generally expand various existing databases to support batch execution task scenarios such as machine learning, confidential computing, and heterogeneous computing with high performance. At the same time, since multiple tuples are saved in the order provided to the target computing node, the execution order of the data in the tuples remains unchanged. Moreover, the solution of this embodiment belongs to a plug-in expansion of the original database computing solution, without modifying the original nodes and computing engines of the database, and is easy to iterate and upgrade with the product. When the database performance is improved, it will not affect its stability.

[0095] As can be seen from the above description, the query statement can be sent by the user through the user terminal. As shown in FIG. 4, a schematic diagram of scenario interaction in an actual application of the embodiments of the present application is shown.

[0096] The user can send a query request to the server 402 through the user terminal 401, and the query request may include a query statement.

[0097] Among them, the user terminal can be a browser, an APP (Application), a web application such as an H5 (HyperText Markup Languages, the 5th edition) application, a light application (also known as a small program, a lightweight application program), or a cloud application, etc. The user terminal can be deployed in an electronic device and needs to rely on the device or certain apps in the device to run, etc.

[0098] The server can include servers providing various services, such as a server for processing the query request sent by the user terminal. It should be noted that the server can be implemented as a distributed server cluster composed of multiple servers, or can be implemented as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology.

[0099] A database engine is deployed in the server, and the database engine can obtain and parse the query statement, generate an execution plan for the query statement, and add a target computing node between the first computing node and the second computing node of the execution plan. After that, the execution plan is executed, so that the first computing node provides the obtained tuple as an output result to the target computing node, and the tuple provided by the first computing node is saved in the target computing node, and the saved multiple tuples are batch processed according to the target operation, and the multiple output results obtained by the batch processing are respectively provided to the second computing node. The second computing node can continue to process the multiple output results and provide the output results obtained by the processing to its parent node, etc. After the execution plan is executed, the query result corresponding to the query statement can be obtained. After that, the server 402 can feed back the query result corresponding to the execution plan to the user end 401.

[0100] By adding the target computing node as the parent node of the computing node that executes the target computing operation in the execution plan, the tuple obtained by the first computing node can be output to the target computing node for storage, and when the tuple storage capacity of the target computing node meets the processing requirements, multiple tuples are batch processed, and multiple output results are provided to the second computing node, that is, the parent node of the target computing node. Compared with the traditional solution in which the computing node accepts a tuple as input for calculation and generates a tuple as output, the one-by-one calculation method realizes batch execution, simplifies the calculation process, and can better handle computing tasks in scenarios involving a large number of tuple calculations, expanding the applicable scenarios of computing task execution.

[0101] FIG5 shows a structural diagram of an embodiment of a data processing device provided by the present application. The device may include an acquisition module 501 , a generation module 502 and an execution module 503 .

[0102] The acquisition module 501 can be used to acquire a query statement.

[0103] The generation module 502 can be used to generate an execution plan for the query statement and add a target computing node between a first computing node and a second computing node in the execution plan; wherein the first computing node is a computing node that performs a target computing operation; the target computing node is used as a parent node of the first computing node; and the second computing node is used as a parent node of the target computing node.

[0104] The execution module 503 can be used to execute the execution plan to provide the obtained tuples as output results to the target computing node at the first computing node; and save the tuples provided by the first computing node at the target computing node, batch process the saved multiple tuples according to the target operation, and provide the multiple output results obtained from the batch process to the second computing node respectively.

[0105] In some embodiments, the generation module 502 can be specifically used to generate an execution plan for the query statement, and add a target computing node between the first computing node and the second computing node in the execution plan when the query statement involves a target operation that meets preset conditions.

[0106] In some embodiments, the generation module 502 can be specifically used to detect that the query statement involves a target operation; rewrite the query statement to apply a target execution function outside the operation expression corresponding to the target operation; parse the query statement to generate an execution plan, and add a target computing node between the first computing node and the second computing node in the execution plan when the query statement includes the target execution function.

[0107] In some embodiments, the target operation includes an operation for generating custom type data according to a pre-configured custom type and the operation requirements of the custom type;

[0108] The execution module 503 can be specifically used to determine that the operation result passes the verification when executing the target operation on the obtained tuples in the first computing node, save the data participating in the target operation as a data object into the tuples, and provide the tuples as output results to the target computing node; in the target computing node, batch process the data objects in the saved multiple tuples according to the target operation to generate multiple operation results of the custom type; convert the multiple operation results into corresponding original data types respectively; determine multiple output results to be provided to the second computing node according to the multiple operation results.

[0109] In some embodiments, the device may further include a storage module for storing the multiple output results in a processing order for the second computing node to obtain the multiple output results in sequence.

[0110] In some embodiments, the storage module may be specifically configured to sequentially place the multiple output results into a storage queue O in the order in which the corresponding tuples are provided to the target computing node.

[0111] In some embodiments, the data participating in the target operation includes original digital type data, and the target operation includes a digital operation and / or comparison operation for generating custom type data according to a preconfigured custom type and the operation requirements of the custom type.

[0112] The execution module 503 may be specifically configured to, in the target computing node, when the target operation includes a digital operation for generating custom type data according to a preconfigured custom type and the operation requirements of the custom type, batch process the data objects in the multiple saved tuples according to the digital operation to generate operation results of a custom digital type; when the target operation includes a comparison operation for generating custom type data according to a preconfigured custom type and the operation requirements of the custom type, batch process the data objects in the multiple saved tuples according to the comparison operation to generate operation results of a custom boolean type; when the target operation includes a digital operation and a comparison operation for generating custom type data according to a preconfigured custom type and the operation requirements of the custom type, batch process the data objects in the multiple saved tuples by sequentially performing the digital operation and the comparison operation in the order of operations. When the last operation is a digital operation, generate operation results of a custom digital type, and when the last operation is a comparison operation, generate operation results of a custom boolean type; and when the operation results include operation results of a custom digital type, convert the operation results of the custom digital type into operation results of an original digital type; when the operation results include operation results of a custom boolean type, convert the operation results of the custom boolean type into operation results of an original boolean type.

[0113] In some embodiments, the database corresponding to the query statement is a ciphertext database, and the data participating in the target operation includes ciphertext data type data.

[0114] The execution module 503 may be specifically configured to call external hardware to decrypt the data objects in the multiple saved tuples, batch process the decrypted data objects according to the target operation to generate multiple operation data, and encrypt the multiple operation data to obtain multiple operation results of a custom type.

[0115] In some embodiments, the obtaining module 501 may be specifically configured to receive a query statement sent by a client.

[0116] The apparatus may further include: A sending module, which may be configured to feed back the query result corresponding to the execution plan to the client.

[0117] FIG. 6 shows a schematic structural diagram of a computing device according to an embodiment provided by the present application. The device may include a storage component 601 and a processing component 602.

[0118] The storage component 601 stores one or more computer program instructions, wherein the one or more computer program instructions are called and executed by the processing component 602 to implement the data processing method shown in FIG. 1.

[0119] In practical applications, the computing device may be implemented as a server in the system architecture shown in FIG. 4.

[0120] The processing component may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for executing the above method.

[0121] The storage component is configured to store various types of data to support operations on the terminal. The storage component may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0122] Of course, the above computing device may necessarily further include other components, such as an input / output interface, a communication component, etc.

[0123] The input / output interface provides an interface between the processing component and the peripheral interface module, and the above peripheral interface module may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices, etc.

[0124] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, which can implement the data processing method shown in FIG. 1 when executed by a computer. The computer-readable medium may be included in the computing device described in the above embodiments; or it may exist alone without being assembled into the computing device.

[0125] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above.

[0126] The embodiments of the present application also provide a computer program product storing a computer program, which can implement the data processing method shown in FIG. 1 when executed by a computer.

[0127] In such an embodiment, the computer program may be downloaded and installed from a network and / or installed from a removable medium. When the computer program is executed by a processor, it executes various functions defined in the system of the present application.

[0128] It should be noted that the above computing device may be a physical device or an elastic computing host provided by a cloud computing platform etc. It may be implemented as a distributed cluster composed of multiple servers or terminal devices, or may be implemented as a single server or a single terminal device.

[0129] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0130] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0131] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

Claims 1. A data processing method, wherein, Including: Obtaining a query statement; Generating an execution plan for the query statement, and adding a target computing node between a first computing node and a second computing node in the execution plan; wherein, the first computing node is a computing node for performing a target operation; the target computing node is used as the parent node of the first computing node; the second computing node is used as the parent node of the target computing node; executing the execution plan to provide the obtained tuples as output results to the target computing node at the first computing node; saving the tuples provided by the first computing node at the target computing node, batch-processing the saved multiple tuples according to the target operation, and respectively providing the multiple output results obtained from the batch processing to the second computing node.

2. The method according to claim 1, wherein The generating the execution plan for the query statement and adding a target computing node between the first computing node and the second computing node in the execution plan includes: generating the execution plan for the query statement, and adding a target computing node between the first computing node and the second computing node in the execution plan when the query statement involves a target operation that meets a preset condition.

3. The method according to claim 2, wherein The preset condition includes one or more of the following implementation manners: an operation for a specified data type; an operation that meets a predetermined operation requirement; an operation for calling external hardware; and an operation for generating custom type data according to a pre-configured custom type and the operation requirement of the custom type.

4. The method according to claim 1, wherein The generating the execution plan for the query statement and adding a target computing node between the first computing node and the second computing node in the execution plan includes: detecting that the query statement involves a target operation; rewriting the query statement to apply a target execution function to the outer layer of the operation expression corresponding to the target operation; parsing the query statement to generate an execution plan, and adding a target computing node between the first computing node and the second computing node in the execution plan when the query statement includes the target execution function.

5. The method according to claim 1, wherein The target operation includes an operation for generating custom type data according to a pre-configured custom type and the operation requirement of the custom type; the providing the obtained tuples as output results to the target computing node at the first computing node Includes: in the first computing node, when determining that the operation result passes the verification for the obtained tuples when performing the target operation, saving the data participating in the target operation as a data object to the tuples, and providing the tuples as output results to the target computing node; The batch processing of the saved multiple tuples according to the target operation includes: in the target computing node, batch processing the data objects in the saved multiple tuples according to the target operation to generate multiple operation results of the custom type; respectively converting the multiple operation results into corresponding original data types; and determining multiple output results for providing to the second computing node according to the multiple operation results.

6. The method according to claim 1, wherein The method further includes: storing the multiple output results in the processing order for the second computing node to sequentially obtain the multiple output results.

7. The method according to claim 6, wherein, The storing the multiple output results in the processing order includes: sequentially putting the multiple output results into a storage queue according to the order in which the corresponding tuples are provided to the target computing node; and the storage queue is a circular buffer or a linked list structure.

8. The method according to claim 1, wherein The batch processing of the data objects in the saved multiple tuples according to the target operation includes: in the target computing node, calling external hardware to batch process the data objects in the saved multiple tuples according to the target operation.

9. The method according to claim 5, wherein The data participating in the target operation includes original digital type data, and the target operation includes digital operation and / or comparison operation for generating custom type data according to a pre-configured custom type and the operation requirements of the custom type; the batch processing of the data objects in the saved multiple tuples in the target computing node according to the target operation to generate multiple operation results of the custom type includes: in the target computing node, when the target operation includes a digital operation for generating custom type data according to a pre-configured custom type and the operation requirements of the custom type, batch processing the data objects in the saved multiple tuples according to the digital operation to generate operation results of the custom digital type; when the target operation includes a comparison operation for generating custom type data according to a pre-configured custom type and the operation requirements of the custom type, batch processing the data objects in the saved multiple tuples according to the comparison operation to generate operation results of the custom boolean type; when the target operation includes a digital operation for generating custom type data according to a pre-configured custom type and the operation requirements of the custom type In the case of performing digital arithmetic operations and comparison operations on custom type data according to requirements, for the data objects in the multiple saved tuples, perform the digital arithmetic operations and comparison operations in sequence according to the operation order for batch processing. In the case where the last arithmetic operation is a digital arithmetic operation, generate an arithmetic result of a custom digital type, and in the case where the last arithmetic operation is a comparison operation, generate an arithmetic result of a custom boolean type; the converting the multiple arithmetic results into corresponding original data types respectively includes: in the case where the arithmetic results include arithmetic results of a custom digital type, converting the arithmetic results of the custom digital type into arithmetic results of an original digital type; in the case where the arithmetic results include arithmetic results of a custom boolean type, converting the arithmetic results of the custom boolean type into arithmetic results of an original boolean type.

10. The method according to claim 5, wherein The database corresponding to the query statement is a fully homomorphic encrypted database, and the data participating in the target arithmetic operation includes data of a ciphertext data type; The batch processing of the data objects in the multiple saved tuples according to the target arithmetic operation to generate multiple arithmetic results of the custom type includes: calling a decryption component to decrypt the data objects in the multiple saved tuples, and batch processing the decrypted data objects according to the target arithmetic operation to generate multiple arithmetic data, and encrypting the multiple arithmetic data to obtain multiple arithmetic results of the custom type.

11. The method according to claim 1, wherein, The obtaining the query statement includes: receiving a query statement sent by a client; the method further includes: feeding back the query result corresponding to the execution plan to the client.

12. A computing device, wherein, It includes a storage component and a processing component; the storage component stores one or more computer program instructions, the computer program instructions are called and executed by the processing component, and the processing component executes the one or more computer program instructions to implement the data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, wherein, It stores a computer program, and the computer program is executed by a computer to implement the data processing method according to any one of claims 1 to 11.

14. A computer program product, wherein, It stores a computer program, and when the computer program is executed by a computer, it implements the data processing method according to any one of claims 1 to 11.

Images