Computation interface and implementation method based on embedded database

Abstract

The present invention discloses a computing interface based on an embedded database and an implementation method thereof. By using the encapsulation type method, the encapsulated data can be efficiently operated by calling the interface in Python. Only one data conversion is required during input and output, and there is no need to repeatedly convert the data into Python objects during calculation, thereby avoiding the inconsistency of data types and improving the overall computing efficiency. Through the SQL execution system and the database self-defined function capability, the self-defined function of the dynamic language Python is converted into the self-defined function of the embedded database through translation. During calculation, there is no need to rely on Python for real-time parsing, and the database kernel can be directly used for efficient calculation. By providing a multi-threaded context management method, the context information required for each function call is hidden, thereby lowering the user usage threshold.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and in particular relates to a computing interface based on an embedded database and an implementation method thereof. Background Art

[0002] In recent years, high-performance databases have gradually become the core of modern information systems. However, traditional databases and traditional analysis tools have exposed some limitations that are difficult to ignore when dealing with complex scenarios. These limitations are mainly reflected in the inconsistency of data types, computing efficiency, and context management. Taking Python as an example, the inconsistency of data types will lead to repeated conversion of Python objects during calculations; in dynamic languages ​​such as Python, the efficiency of user-defined functions is often poor, and they need to rely on dynamic languages ​​such as Python for real-time parsing, which reduces computing efficiency; high-performance databases usually require computing acceleration based on multi-threading. In user-friendly languages ​​such as Python, if the context management part is exposed to users, it will increase the complexity of the user interface. Summary of the invention

[0003] In order to solve the shortcomings of the existing technology, achieve high-performance computing capabilities, reduce the difficulty of cross-language data interaction in the database, and lower the user threshold, the present invention adopts the following technical solutions:

[0004] Computational interfaces based on embedded databases, including encapsulation type interfaces, function interfaces, and query interfaces;

[0005] The encapsulation type interface binds the built-in type of the embedded database kernel to the client language by binding the type. During processing, the client language only stores data references and accesses internal data through the exposed interface.

[0006] The function interface includes binding functions, custom functions and partial applications; the binding function binds the preset built-in function in the embedded database kernel to the client through the type system; the custom function includes callback, script construction and translation, the callback is to convert the client-defined function into a FunctionDef function definition object that can be recognized by the embedded database kernel, and the script construction and translation directly use the embedded database kernel, define the function object through the script and convert the function object into a binding object and return it to the client language; the partial application maps the parameters of the built-in function to the client language and manages the parameters, and is compatible with the method of fixing some parameters of the function to generate a new function and realize partial application of the function;

[0007] The query interface is based on the binding type, executes the query in the embedded database kernel through type conversion and obtains the query result.

[0008] Furthermore, the binding function maps the built-in function of the embedded database kernel (usually an instance of FunctionDef) to the instance object of the client through type binding, and keeps the function's properties (having function parameter prompts and supporting function call operators). The process of binding the built-in function is as follows:

[0009] Get the instance FuncA of the built-in function object FunctionDef, convert the instance FuncA into a binding object BindFuncA, and obtain its syntax definition through the embedded database kernel;

[0010] Implement the pseudo function funcA in the client language according to the grammatical definition. The pseudo function funcA has no specific implementation but only parameter definition. Define a conversion method in the client language.

[0011] Combining the syntax hint and name of the pseudo function funcA with the function call operator of the binding object BindFuncA;

[0012] During the conversion process, the parameter definition of the pseudo function funcA in the client language is obtained and saved in the instance FuncA.

[0013] Furthermore, the process of calling a bound function in the client language is as follows:

[0014] Call the provided function interface in the client language and pass in the function parameters;

[0015] Convert the function parameter to a Constant.

[0016] According to the parameter definition of the pseudo function funcA in the function binding process, fill the function parameters with default values ​​and determine the actual corresponding parameter positions;

[0017] Get the current context;

[0018] Based on the embedded database kernel, obtain the context and function parameters and execute the function;

[0019] The function execution return result Constant is obtained, the type conversion module is called, and the result is converted into the corresponding binding type BindObject and returned to the client language.

[0020] Furthermore, the process of defining and calling a custom function through the callback is as follows:

[0021] Based on the FunctionDef class in the embedded database kernel, the inherited BindFunctionDef class is implemented, which can save the callback of the client-defined function;

[0022] Implement a conversion method that can obtain the custom function parameter information of the client language, and create an instance of the corresponding inherited BindFunctionDef class to store the parameter information in it;

[0023] When calling a custom function in the client language, the inherited instance of the BindFunctionDef class is called according to the steps of calling a bound function in the client language until the current context Context is obtained, the client function bound in the inherited instance of the BindFunctionDef class is executed, the obtained parameter information is converted into a binding object and then passed into the client function for execution, and the client function execution result is converted into a Constant to obtain the execution result;

[0024] When a custom function is called in the embedded database kernel, the current context provided by the kernel is placed on the top of the context stack of the current thread in the session context management module, the parameter information is converted into a client binding object, the client callback function saved in the inherited BindFunctionDef class is passed in, the callback function is executed, the placed context is popped out, and the execution result is returned to the embedded database kernel.

[0025] Furthermore, the process of defining a custom function through the script construction is as follows:

[0026] Pass the definition of the custom function into the script construction interface;

[0027] Get the current context from the session context management module;

[0028] According to the current context Context and construction script, the embedded database kernel module interface is passed in to obtain the returned function definition FunctionDef object;

[0029] Obtain parameter information of the current custom function according to the internal interface, and save it in the FunctionDef object;

[0030] Convert the FunctionDef object into a binding object and return it to the client language;

[0031] The process of defining a custom function through the translation is as follows:

[0032] Define custom functions in the client language;

[0033] The embedded database kernel calls the conversion method and passes it into the custom function;

[0034] Get the source code of the custom function by calling the client language reflection method;

[0035] Translating the source code into a scripting language supported by the database system according to the syntax tree;

[0036] The translated script is executed according to the steps of the script structure to define the custom function.

[0037] Further, the process of executing the script in the client language is as follows:

[0038] The client calls the encapsulated execution interface exec and passes the script to be executed to the embedded database kernel, so that it calls the session context management module, obtains the current context Context, and receives the context Context and the script to be executed script through the kernel execution interface execute, executes the script and returns the execution result Constant. The system calls the toBindObject method provided by the data type conversion module to convert the execution result Constant to a data reference ConstantRef and save it in the binding type BindObject. The execution interface exec returns the data object corresponding to the binding type BindObject in Python.

[0039] Furthermore, the process of querying through the query interface is as follows:

[0040] Get the query statement and variable dictionary;

[0041] Convert the data in the variable dictionary to Constant;

[0042] Get the current context;

[0043] The query statement, the variable dictionary after type conversion, and the context Context are passed into the embedded database kernel to execute the query;

[0044] Convert the execution result to the binding type through type conversion;

[0045] Returns a binding type object to the client language.

[0046] The computing interface realization method based on embedded database adopts a user interface and an embedded database kernel module, a type conversion module and a context management module arranged in cooperation therewith, wherein the user interface is the computing interface based on embedded database;

[0047] The embedded database kernel module executes the incoming tasks and directly processes the tasks through the type system and the kernel execution interface, the type system provides the built-in types and presets the built-in functions; the kernel execution interface corresponds to the function interface and the query interface;

[0048] The type conversion module is used for data type binding and conversion, converting the data type of the client language to the type system (toConstant(hint)), and converting the database type to the data type of the client language (toClientObject(hint));

[0049] The context management module obtains corresponding information from the context according to usage in a multi-threaded computing environment through the context stack to ensure the correctness of the context information. The context information is the data required for the embedded database kernel module to perform any task (SQL, calculation function, etc.). The tasks performed by the client need to obtain the context for execution.

[0050] Furthermore, based on the logical relationship between the data structures of the embedded database kernel module, a parent class ConstantRef (constant reference) of all data types is defined to represent a reference to a data object. A constant Constant is stored in each parent class ConstantRef. For all instances of the binding object BindObject inherited from the parent class ConstantRef, they correspond to instances in the client language.

[0051] The process of using the type conversion module in the client language (Python) is as follows:

[0052] Call the function interface to obtain the Python object of the client language;

[0053] The toConstant function provided by the type conversion module determines whether it is an encapsulation type through the client language Python type inheritance relationship. If it is an encapsulation type BindObject, the constant Constant stored in the binding object BindObject of the encapsulation type is directly retrieved; otherwise, it checks whether there is a type hint. If so, it converts according to the hint. If not, it converts according to the default mode to obtain the constant Constant;

[0054] The obtained constant Constant is passed to the embedded database kernel module to execute the task (execute script, call function, etc.) and obtain its execution result Constant;

[0055] The data form and data type of the execution result Constant are determined, and converted into a corresponding binding object BindObject, so that the function interface feeds back the binding object BindObject to the client.

[0056] From the perspective of passing in parameters when calling a function, in most scenarios, users do not want to directly perceive the data of the database type system in the client language environment. For example, when executing the function add(a, b), users prefer to use the native type of the client language, such as add(1, 2), rather than using the encapsulation type: a=Bind(1); b=Bind(2); add(a, b). In the present invention, users can be unaware of the database type in most scenarios. In special cases, such as when the automatic type conversion does not meet expectations and the parameter type needs to be specified, a type hint can be provided to the toConstant function to indicate which rules the system should follow for conversion, so as to obtain the expected type conversion result.

[0057] On the other hand, from the perspective of the execution result obtained by calling the function, the present invention will directly return the binding type BindObject. The advantage of this processing is that the data obtained by the user can retain information to the greatest extent. When the user needs to convert to a specific client language data, he can manually call the toClientObject function.

[0058] Furthermore, the context management module includes a context stack and a group of interfaces for operating the context stack. One thread corresponds to a unique context stack. The context stack interface includes a context push interface, a context pop interface, and a context top interface. During the execution of concurrent queries, the kernel execution stack and the client execution stack in the thread are switched. During the switching process, the created context Context is pushed to the top of the context stack through the context push interface, and the context Context is obtained through the context top interface. After the execution is completed, the context Context is popped out through the context pop interface.

[0059] The advantages and beneficial effects of the present invention are:

[0060] The present invention automatically maps the data types in the database with dynamic programming language objects through data type mapping and conversion, thereby realizing seamless data transfer between calculations. Because the objects actually operated on the Python side are only the mappings of internal data, there is no need to frequently perform type conversions between multiple calculations, and only conversions are required in the input and output links, thereby improving the overall execution efficiency of the system; the present invention encapsulates the built-in functions of the function library and the user-defined functions into a unified function object through function binding and parameter management, and provides complete parameter prompts in Python through parameter management and other methods, thereby lowering the user's usage threshold and unifying the use of function objects and other objects; the present invention provides a variety of user-defined function methods through diversified user-defined function support. If the program Part of the logic relies on Python libraries to complete, and the callback mode can be used to define functions. The powerful various Python libraries can be used to quickly implement services during use. If the user is familiar with the database scripting language, the function can be defined by directly writing the script, and the execution efficiency is consistent with the way the script is executed in the database. If the user wants to retain the execution efficiency and also wants to implement it with Python logic, the script that conforms to Python syntax can be converted into a database script through the translation mode. The present invention hides the context conditions of function execution through multi-threaded session context management, lowers the usage threshold, and ensures that the execution context will not be confused when using multiple threads in various scenarios by using a context stack for each thread. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a diagram of the overall architecture of the computing interface and implementation based on the embedded database in the embodiment of the present invention.

[0062] Figure 2 It is a flow chart of type conversion in an embodiment of the present invention.

[0063] Figure 3a This is a diagram of a multi-threaded context session management system.

[0064] Figure 3b It is a schematic diagram of multi-threaded context session management in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0066] like Figure 1As shown, the computing interface implementation system based on the embedded database includes a user interface and an embedded database kernel module, a type conversion module (type system) and a session context management module that are configured in conjunction with the user interface, and the user interface includes an encapsulation type interface, a function interface and a cross-language SQL query interface.

[0067] Embedded database kernel module: Responsible for the core processing capabilities, it mainly provides two capabilities: (1) executing incoming scripts, functions, etc.; (2) providing a rich type system that can directly use the type system to process tasks instead of relying entirely on script execution and function calls.

[0068] Specifically, the embedded database kernel module mainly provides a kernel data type system and an external kernel execution interface.

[0069] In the type system, all classes that can be used by users are defined based on Constants, and provide built-in data types such as MetaCode and FunctionDef.

[0070] There are two main types of external kernel execution interfaces. One is for function interfaces, such as the execute function that can execute scripts. Its function prototype is Constant execute(ctx: Context, script: str). This function can execute the corresponding script in the context and return the execution result. The other is for SQL query interfaces.

[0071] Note: In actual implementation, Context may have multiple forms, but its main purpose is to save the execution context and provide a running environment, so it is simplified to Context.

[0072] The steps for a user to execute a script in a client language (the programming language actually used by the user, which is Python in the embodiment of the present invention) are as follows:

[0073] (1) The client calls the encapsulated exec interface and passes in the script to be executed;

[0074] (2) The system calls the session context management module to obtain the current context;

[0075] (3) The kernel execution interface execute receives the context Context and the script to be executed script;

[0076] (4) Kernel execution;

[0077] (5) The kernel returns the execution result Constant;

[0078] (6) The system calls the toBindObject method provided by the data type conversion module to convert the Constant returned in step (5) into a data reference ConstantRef and save it in the binding type BindObject;

[0079] (7) The exec interface returns the data object corresponding to the binding type BindObject in Python.

[0080] Encapsulation type interface: By implementing binding types, the type system provided by the kernel is bound to the client language to achieve a one-to-one correspondence. During processing, the client language (such as Python) only stores data references and accesses internal data through the exposed interface.

[0081] Type conversion module: used for data type binding and conversion, such as Figure 2 As shown, based on the logical relationship between the data structures of the kernel module, the parent class ConstantRef (constant reference) of all data types is defined to represent a reference to a data object. Each ConstantRef stores a Constant (constant). In the following, BindObject will be used to represent all binding objects inherited from ConstantRef. An instance of a binding object corresponds to an instance in the client language.

[0082] This module mainly provides two types of methods. One is responsible for converting the data type of the client language to the database type system (toConstant(hint)); the other is responsible for converting the database type to the data type of the client language as needed (toClientObject(hint)).

[0083] The steps for users to use type conversion in the client language (Python) are as follows:

[0084] (1) The client calls the function interface and passes in the corresponding Python object;

[0085] (2) The toConstant function provided by the system call type conversion module;

[0086] (3) Determine whether it is an encapsulation type through the Python type inheritance relationship. If it is an encapsulation type BindObject, execute step (4), otherwise execute step (5); the encapsulation type refers to the binding instance of the database internal object stored inside it. There is a one-to-one correspondence between the instance and the type in the client language. The internal object refers to the object in the type system implemented by the internal language of the database system;

[0087] (4) Directly retrieve the Constant stored in BindObject;

[0088] (5) Check whether there is a type hint. If there is a type hint, convert it according to the hint. Otherwise, convert it according to the default mode and finally get Constant;

[0089] (6) The Constant obtained in step (4) or step (5) is passed to the kernel module to execute the task (execute the script, call the function, etc.), and the execution result Constant is obtained;

[0090] (7) Determine the data format and data type of Constant and convert it to the corresponding BindObject;

[0091] (8) The client obtains the execution result BindObject through the function interface in step (1).

[0092] From the perspective of passing in parameters when calling a function, in most scenarios, users do not want to directly perceive the data of the database type system in the client language environment. For example, when executing the function add(a,b), users prefer to use the native type of the client language, such as add(1, 2), rather than using the encapsulation type: a=Bind(1);b=Bind(2);add(a,b). Under this design, users can be unaware of the database type in most scenarios. In special cases, such as when the automatic type conversion does not meet expectations and the parameter type needs to be specified, a type hint can be provided to the toConstant function to indicate which rules the system should follow for conversion, so as to obtain the expected type conversion result.

[0093] On the other hand, from the perspective of the execution result obtained by calling the function, the present invention will directly return the binding type BindObject. The advantage of this processing is that the data obtained by the user can retain information to the greatest extent. When the user needs to convert to a specific client language data, he can manually call the toClientObject function.

[0094] Function interface: The encapsulation type interface only defines the type, while the function interface includes bound functions, partial applications, and custom functions. The bound function is to bind the preset built-in function in the kernel to the client through the type system; the custom function implements the user-defined function in a specific way; the partial application maps the parameters of the built-in function to the client language, usually fixing some parameters of a function to generate a new function. The new function object is no different from an ordinary function.

[0095] Bind function: Map the database kernel's calculation function (usually an instance of FunctionDef) to the client's instance object through type binding, and keep the function's properties (having function parameter prompts and supporting function call operators).

[0096] The steps to bind a built-in function are as follows:

[0097] (1) Get the instance FuncA of the internally implemented function object FunctionDef;

[0098] (2) Convert the instance FuncA to the binding object BindFuncA;

[0099] (3) Obtain its syntax definition through the kernel;

[0100] (4) Implement the pseudo function funcA in the client language according to the grammatical definition. funcA has no specific implementation, only parameter definition;

[0101] (5) Define a conversion method in the client language to combine the syntax hint and name of the pseudo function funcA with the function call operator of the binding object BindFuncA;

[0102] (6) During the conversion process, the parameter definition of the pseudo function funcA in the client language is obtained and saved in the instance FuncA.

[0103] There are usually two situations for calling a bound function: 1. Calling the function in the client language, 2. Calling the function by the kernel during execution in the kernel. Generally speaking, when calling a function in the kernel, the kernel will be responsible for handling the context, so only the scenario of calling the function in the client language is introduced.

[0104] The steps to call a bound function in the client language are as follows:

[0105] (1) Call the provided function interface in the client language and pass in the function parameters;

[0106] (2) The system calls the toConstant function provided by the type conversion module (step (2) of the type conversion process), processes the passed parameter, and converts it into a Constant;

[0107] (3) According to the parameter definition of funcA saved in the function binding process (step (6) of binding built-in functions), fill the incoming parameters with default values ​​and determine the actual corresponding parameter positions;

[0108] (4) The system calls the session context management module to obtain the current context;

[0109] (5) Call the kernel module, pass in the context and function parameters, and execute the function;

[0110] (6) Get the function execution return result Constant, call the type conversion module, convert it to the binding type BindObject and return it to the client language.

[0111] Custom function module: mainly divided into three methods: callback, script construction and translation. Callback allows users to write any client-defined function, which is converted into a FunctionDef object that can be recognized by the kernel module through the conversion method. Script construction and translation both directly use the kernel module's script execution capability to define function objects through scripts, and then convert the generated function objects into binding objects and return them to the client language.

[0112] Callback:

[0113] Define a custom function through callback and call it as follows:

[0114] (1) Inherit from the FunctionDef class in the kernel module and implement the BindFunctionDef class, which can save the callback of the client user-defined function;

[0115] (2) Implement a conversion method that can obtain function callbacks and syntax information of the client language;

[0116] (3) The user implements a custom function in the client language and converts it using the conversion method provided in step (2);

[0117] (4) The conversion method parses the parameter name, parameter type and other information of the user-defined function, creates a corresponding BindFunctionDef instance, and stores the information in it;

[0118] (5) When calling the function, there are two cases: the first is to call the function in the client language and execute step (6); the second is to call the function in the kernel and execute step (7);

[0119] (6) The BindFunctionDef instance in step (4) calls a bound function in the client language (executes to step (4) of calling the bound function); executes the client function bound in the BindFunctionDef instance, and then converts the obtained parameters into a binding object through step (3) and passes them into the client function for execution; the client function execution result is converted into a Constant, and then ends;

[0120] (7) Put the current context provided by the kernel into the top of the context stack of the current thread in the session context management module; convert the parameters into the client binding object and pass it into the client callback function saved in step (4) BindFunctionDef; execute the callback function; pop the context put in; return the result of executing the callback function to the kernel module, and end.

[0121] Script construction:

[0122] The execution process of a custom function constructed by scripting is the same as that of a bound function. The definition process is as follows:

[0123] (1) Pass the function definition into the script construction interface;

[0124] (2) Obtain the current context from the session context management module;

[0125] (3) According to the current context Context and construction script, pass in the kernel module interface and obtain the returned function definition FunctionDef object;

[0126] (4) Obtain parameter information of the current function according to the internal interface and save it in FunctionDef;

[0127] (5) The FunctionDef in step (4) is converted into a binding object and returned to the client language.

[0128] translate:

[0129] Similar to the script construction, the execution process is also consistent with the binding function. The definition process is as follows:

[0130] (1) The user defines a custom function in the client language;

[0131] (2) Call the conversion method and pass it into the custom function;

[0132] (3) The system obtains the source code of the custom function by calling the client language reflection;

[0133] (4) Translate it into a scripting language supported by the database system based on the syntax tree;

[0134] (5) Execute the script translated in step (4) according to the definition steps of the script construction method.

[0135] Session context management module: Ensure the correctness of context information in a multi-threaded computing environment through the context stack. Context information is the data required for the kernel module to execute any task (SQL, calculation function, etc.). Tasks executed by the client need to obtain the context for execution.

[0136] The present invention designs a multi-threaded context session management method. When an embedded database kernel performs a computing task, it is usually necessary to retain and obtain information based on a stack (Heap) and a session (Session). If the session and other interfaces are directly exposed to the user, it will cause unnecessary user usage thresholds. Therefore, a context (Context) is defined to encapsulate the stack (Heap) and the session (Session), and the corresponding information can be obtained from the context according to the usage.

[0137] The present invention designs a multi-threaded context management method. The module mainly includes a unique context stack of a thread, and a series of interfaces for operating the context stack: push (push context) / pop (pop context) / top (get top context). Each context object provides an interface for obtaining the current Heap (kernel module execution context): getHeap. Among them, different context objects can store different data types, some only store Heap; some store session, and get Heap from session object when needed. When each thread (including the thread created by the client language and the execution thread of the kernel module) is created, a context stack that is only used by the same thread will be created, and its life cycle is consistent with the thread.

[0138] Consider the following scenario: In the client language, a custom function is created using the callback pattern and a built-in function is used in the callback. The custom function is then used to execute concurrent queries in the provided query interface.

[0139] The pseudo code is as follows:

[0140] @udf

[0141] Def myAdd(x, y){

[0142] Return F.add(x, y)

[0143] }

[0144] Sql("SELECT myAdd(a,b) FROM table")

[0145] The pseudo code uses the callback mode to create a custom function myAdd, which calls the built-in function F.add to calculate and return the result. When executing this concurrent query, the kernel will create multiple execution threads and execute the myAdd function on the data in each thread.

[0146] If the session context management module is not used, the execution flow is as follows Figure 3a As shown:

[0147] (1) In the client language, call the interface to execute the concurrent query;

[0148] (2) Enter the kernel language execution stack from the client language execution stack through the connection, and the kernel provides kernel context Heap 1 to execute the UDF function BindFunctionDef;

[0149] (3) Switch from the kernel language execution stack to the client language execution stack and execute the client language callback (myAdd) saved in the UDF function;

[0150] (4) Execute the built-in function F.add in the callback function;

[0151] (5) Switch from the client language execution stack to the kernel language execution stack and execute the corresponding FunctionDef;

[0152] At this time, when FunctionDef is executed, it will not be able to obtain the current kernel context, because the runtime does not always remain in the kernel language execution stack. After switching to the client language execution stack, the current kernel context cannot be retained.

[0153] If the session context management module is used, the execution flow is as follows Figure 3b As shown:

[0154] (1) The client executes a concurrent query in thread 1. At this time, there are two elements in the context stack: the default context at the bottom of the stack and connection A at the top of the stack. The concurrent query will be executed in connection A;

[0155] (2) During kernel processing, multiple threads such as thread 2 will be created;

[0156] (3) When the kernel processes and calls UDF (BindFunctionDef), the kernel context is Heap1;

[0157] (4) Use Heap1 to create a context Context and push it into the context stack;

[0158] (5) Switch from the kernel language execution stack to the client language execution stack and execute the callback myAdd;

[0159] (6) Call the built-in function F.add in the client language execution stack;

[0160] (7) Switch from the client language execution stack to the kernel language execution stack;

[0161] (8) Get the current context Context from the top of the context stack and get Heap1 from it;

[0162] (9) Use Heap1 to call FunctionDef. The state of Heap1 still remains the state obtained from the kernel in step (3).

[0163] Cross-language SQL query interface: Usually SQL query needs to import data into the database before execution, which has a certain conversion overhead. The present invention supports direct query of Table encapsulation types or DataFrame objects existing in Python.

[0164] The specific steps for querying using the SQL query interface are as follows:

[0165] (1) The user calls the SQL query interface provided by the system in the client language and passes in the SQL query statement and variable dictionary;

[0166] (2) The system calls the type conversion module to convert the data in the variable dictionary into Constants;

[0167] (3) Call the session context management module to obtain the current context;

[0168] (4) Call the interface in the kernel module and pass in the Context of step (3), the SQL query of step (1), and the variable dictionary converted in step (2);

[0169] (5) Execute the query;

[0170] (6) Call the type conversion module to convert the execution result into a binding type;

[0171] (7) Return the binding type object to the client language.

[0172] The following provides a specific embodiment through an example scenario:

[0173] Scenario description: The user wants to perform a multi-threaded SQL query in Python and use a custom function in it. The logic is to perform the (a+b) * 2 operation on two columns of data, and the operation part is implemented using a bound function.

[0174] Step 1: Custom function

[0175] In Python, use Python syntax to define a function, implement the binding function in the function library, and convert it to the binding object corresponding to FunctionDef through the conversion method provided by the system. The specific code is as follows:

[0176] @udf

[0177] Def myAdd(a, b){

[0178] Return F.mul(F.add(a, b), 2)

[0179] }

[0180] In this step, when executed in Python, it will follow the steps of the custom function callback method. In the end, myAdd will be a binding object, corresponding to a BindFunctionDef that saves the custom function callback;

[0181] Step 2: Define the data

[0182] (1) Call the execution script interface provided by the system in Python and pass in the data construction script;

[0183] (2) Follow the steps of executing a script in the kernel and get the returned binding object t, which represents a data table;

[0184] Step 3: Execute the query

[0185] (1) When executing a query, first write the SQL query statement in Python and pass the relevant objects (data tables, custom functions) as variable dictionaries; for example: Sql("SELECT myfunc(a,b) FROM tb", {'myfunc':myAdd, 'tb': t});

[0186] (2) The SQL query statement and variable dictionary of step (1) are provided to the cross-language SQL query interface through the interface to execute the query, and the query is executed until the session context management module is called to obtain the current context Context (i.e., step (3) of the SQL query). At this time, the current context obtained is Context0 (the default context of the thread);

[0187] (3) The kernel executes the SQL query and calls the myAdd function in the internal multithreading;

[0188] (4) Taking the execution in thread 1 as an example, when creating, a context stack including Context0_1 is created according to the session context management module;

[0189] (5) The kernel module calls the myAdd function and executes the steps according to the custom function (callback method). It executes to (callback step (7)) where it puts the current context provided by the kernel into the top of the context stack of the current thread in the session context management module, obtains the current execution context Context1_1 from the kernel, and puts it into the context stack of the current thread.

[0190] (6) Execute the custom function (in the callback step (7)) to convert the parameters into the client binding object, pass it into the client callback function saved in BindFunctionDef, and execute the callback function;

[0191] (7) Because the binding functions F.add and F.mul are called in the custom callback function, the execution steps of the binding function in the client language are executed until (step (4) of calling the binding function) the system calls the session context management module to obtain the current context Context;

[0192] (8) Because the query execution steps (4) to (8) are all performed in thread 1, the session context management module obtains the top of the context stack from thread 1, namely, Context1_1.

[0193] (9) Continue to call the binding function in the client language, execute (call the binding function step (5)) call the kernel module, pass in the Context and function parameters, and execute the function; after the execution is completed, return the result to the query execution step (3), and then execute the custom function (callback step (7)), pop the top of the context stack, that is, Context1_1;

[0194] (10) Execute the custom function (callback step (7)) and return the execution result to the kernel module;

[0195] (11) The kernel module continues to execute the query and returns the execution results returned in the SQL query interface (query step (7)) to the client language.

[0196] The present invention uses a unified function usage method, so that both built-in functions and custom functions can have complete type hints and parameter checks. Encapsulation types are used to unify data transfer between interfaces and coordinate modules. Diverse custom function support ensures performance while improving ease of use. The context management mechanism reduces the user's usage burden and ensures that the correct context information can be used internally in Python or C++ multi-threading.

[0197] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A computing interface implementation method based on an embedded database, including a package type interface, a function interface and a query interface, characterized in that: The encapsulation type interface binds the built-in type of the embedded database kernel to the client language by binding type; The function interface includes binding functions, custom functions and partial applications; The binding function binds the preset built-in functions in the embedded database kernel to the client; Custom functions include callback, script construction and translation. Callback is to convert the client-defined function into a function definition object that can be recognized by the embedded database kernel. Script construction and translation directly use the embedded database kernel to define the function object and convert the function object into a binding object and return it to the client language. Some applications map the parameters of built-in functions to the client language and manage the parameters. The binding function maps the built-in function of the embedded database kernel to the instance object of the client through type binding, and keeps the function's properties. The process of binding the built-in function is as follows: Get an instance of a built-in function object, convert the instance into a binding object, and obtain its syntax definition through the embedded database kernel; Implement the pseudo function in the client language according to the syntax definition, and the pseudo function only has parameter definitions; combining the syntax hint and name of the pseudo-function with the function call operator of the binding object; Obtain parameter definition of the pseudo function in the client language and save it to the instance; The query interface is based on the binding type, executes the query in the embedded database kernel through type conversion and obtains the query result.

2. The method for implementing a computing interface based on an embedded database according to claim 1, characterized in that: The process of calling a bound function in the client language is as follows: Call the provided function interface in the client language and pass in the function parameters; Perform type conversion on function parameters; According to the parameter definition of the pseudo function in the function binding process, fill the function parameters with default values ​​and determine the actual corresponding parameter positions; Get the current context; Based on the embedded database kernel, obtain the context and function parameters and execute the function; The function execution return result is obtained and converted into the corresponding binding type and returned to the client language.

3. The method for implementing a computing interface based on an embedded database according to claim 1, characterized in that: The process of defining and calling a custom function through the callback is as follows: Based on the classes in the embedded database kernel, implement the callback of the inherited class to save the client-defined function; Obtain the custom function parameter information of the client language, and create an instance of the corresponding inherited class, and save the parameter information therein; When calling a custom function in the client language, the instance of the inherited class is called according to the steps of calling a bound function in the client language, the bound client function in the inherited class instance is executed, the obtained parameter information is converted into a binding object and then passed into the client function for execution, and the execution result is obtained; When the custom function is called in the embedded database kernel, the parameter information is converted into a client binding object, passed into the client callback function saved in the inheritance class, the callback function is executed, and the execution result is returned to the embedded database kernel.

4. The method for implementing a computing interface based on an embedded database according to claim 1, characterized in that: The process of defining a custom function through the script construction is as follows: Pass the definition of the custom function into the script construction interface; Get the current context; According to the current context and construction script, the embedded database kernel is passed to obtain the returned function definition object; Get the parameter information of the current custom function and save it in the object; Convert the object into a binding object and return it to the client language; The process of defining a custom function through the translation is as follows: Define custom functions in the client language; The embedded database kernel passes the custom function; Get the source code of the custom function by calling the client language reflection method; Translate the source code into a scripting language supported by the database according to the syntax tree; The translated script is executed according to the steps of the script structure to define the custom function.

5. The method for implementing a computing interface based on an embedded database according to claim 1, characterized in that: The process of executing a script in a client language is as follows: The client calls the encapsulated execution interface and passes the script to be executed into the embedded database kernel so that it can obtain the current context and execute and return the execution result together with the script to be executed. The execution result is converted into a data reference and saved in the binding type. The execution interface returns the data object corresponding to the binding type.

6. The method for implementing a computing interface based on an embedded database according to claim 1, characterized in that: The process of querying through the query interface is as follows: Get the query statement and variable dictionary; Convert the data in the variable dictionary to the same type; Get the current context; The query statement, the variable dictionary after type conversion, and the context are passed to the embedded database kernel to execute the query; Convert the execution result to the binding type through type conversion; Returns a binding type object to the client language.

7. A computing interface system based on an embedded database, characterized in that: A user interface and an embedded database kernel module, a type conversion module and a context management module arranged in cooperation therewith are used, wherein the user interface is the computing interface based on the embedded database as described in claim 1; The embedded database kernel module processes tasks through a type system and a kernel execution interface, the type system provides the built-in type and presets the built-in function; the kernel execution interface corresponds to the function interface and the query interface; The type conversion module is used for data type binding and conversion, converting the data type of the client language into the type system, and converting the database type into the data type of the client language; The context management module obtains corresponding information from the context according to the usage in a multi-threaded computing environment by means of a context stack.

8. The computing interface system based on embedded database according to claim 7, characterized in that: Based on the logical relationship between the data structures of the embedded database kernel module, parent classes of all data types are defined to represent references to data objects. Constants are stored in each parent class. For all instances of binding objects inherited from the parent class implementation, instances in the client language correspond; The process of using the type conversion module in the client language is as follows: Call the function interface to obtain the object of the client language; The type conversion module determines whether it is an encapsulation type through the client language type inheritance relationship. If it is an encapsulation type, the constant stored in the binding object of the encapsulation type is directly retrieved; otherwise, whether there is a type hint is checked. If so, the constant is converted according to the hint. If not, the constant is converted according to the default mode to obtain the constant. The obtained constant is passed to the embedded database kernel module to execute the task and obtain its execution result; The data format and data type of the execution result are determined, and converted into a corresponding binding object, so that the function interface feeds the binding object back to the client.

9. The computing interface system based on embedded database according to claim 7, characterized in that: The context management module includes a context stack and a group of interfaces for operating the context stack. One thread corresponds to a unique context stack. The context stack interface includes a push context interface, a pop context interface and a stack top context interface. During the execution of concurrent queries, the kernel execution stack and the client execution stack in the thread are switched. During the switching process, the created context is pushed to the top of the context stack through the push context interface, and the context is obtained through the get top context interface. After the execution is completed, the context is popped out through the pop context interface.

Images