Data processing method and device of distributed computing engine and electronic equipment

Abstract

The invention relates to a data processing method and device of a distributed computing engine and electronic equipment. The method comprises the steps that under the condition that a structured query statement of a client side is received, the structured query statement is analyzed, a first task and a target task are obtained, the structured query statement comprises a user-defined function, the target task is a task corresponding to the user-defined function, and the first task is a task corresponding to statements except the user-defined function; calling an interface of the target custom function server to enable the target custom function server to execute the target task; target data generated by the target self-defined function server and first data generated by the working node of the distributed computing engine serve as execution results of the structured query statement and are returned to the client side, and the first data are data obtained when the working node of the distributed computing engine executes the first task. According to the method and the device, the technical problem of low processing efficiency of the structured query statement comprising the custom function is solved.

Description

Technical Field

[0001] This application relates to the field of distributed computing, and in particular, to a data processing method, apparatus, and electronic device for a distributed computing engine. Background Art

[0002] In the prior art, as a distributed computing engine, Trino can receive a structured query statement from a client, and then the worker nodes of Trino process the task corresponding to the structured query statement and give the result. However, if the structured query statement of the user includes a custom function, it is necessary to compile the set of custom functions into a Jar package in advance, then copy the Jar package to all nodes of Trino and restart the cluster before Trino can execute the custom function and process the structured query statement of the user.

[0003] The above method has low processing efficiency for structured query statements including custom functions. Summary of the Invention

[0004] This application provides a data processing method, apparatus, and electronic device for a distributed computing engine to solve the technical problem of low processing efficiency for structured query statements including custom functions.

[0005] In a first aspect, this application provides a data processing method for a distributed computing engine, including: when receiving a structured query statement from a client, parsing the structured query statement to obtain a first task and a target task, where the structured query statement includes a custom function, the target task is the task corresponding to the custom function, and the first task is the task corresponding to the statement other than the custom function; calling an interface of a target custom function server to enable the target custom function server to execute the target task; returning the target data generated by the target custom function server and the first data generated by the worker nodes of the distributed computing engine as the execution result of the structured query statement to the client, where the first data is the data obtained by the worker nodes of the distributed computing engine executing the first task.

[0006] In a second aspect, the present application provides a data processing device for a distributed computing engine, including: a parsing module, configured to parse the structured query statement received from a client to obtain a first task and a target task, where the structured query statement includes a custom function, the target task is a task corresponding to the custom function, and the first task is a task corresponding to the statement other than the custom function; a calling module, configured to call an interface of a target custom function server to enable the target custom function server to execute the target task; a sending module, configured to return the target data generated by the target custom function server and the first data generated by a working node of the distributed computing engine as the execution result of the structured query statement to the client, where the first data is data obtained by the working node of the distributed computing engine executing the first task.

[0007] As an optional example, the calling module includes: a calling unit, configured to determine the function type of the custom function; determine the target custom function server from multiple custom function servers according to the correspondence between the function type and the custom function server; and call the interface of the target custom function server to execute the target task.

[0008] As an optional example, the calling unit includes: a calling subunit, configured to call the target custom function server to execute the target task when there is one target custom function server; and select an idle target custom function server to execute the target task when there are multiple target custom function servers.

[0009] As an optional example, the calling module further includes: an identifying unit, configured to receive a custom function set from the custom function servers before determining the target custom function server from multiple custom function servers according to the correspondence between the function type and the custom function server, where the custom function set records the custom functions included in the custom function servers; and identify the types of the custom functions in the custom function set to establish the correspondence between the function type and the custom function server.

[0010] As an optional example, the calling module includes: a searching unit, configured to search for the target data corresponding to the target task in the local cache; prohibit calling the interface of the target custom function server when the target data is found; and call the interface of the target custom function server when the target data is not found.

[0011] As an alternative example, the above parsing module includes: a parsing unit configured to parse the above structured query statement to obtain the client identifier of the above client; and insert the above client identifier into the above target task. When the above target custom function server executes the above target task, it verifies the access authority of the above client through the above client identifier and determines whether to execute the above target task according to the above access authority.

[0012] As an alternative example, the above device further includes: a monitoring module configured to monitor the size of the total storage space and the available storage space of each custom function server; and expand the above current custom function server when the ratio of the size of the available storage space to the size of the total storage space of the current custom function server is less than a predetermined threshold.

[0013] In a third aspect, the present application provides an electronic device, including: at least one communication interface; at least one bus connected to the above at least one communication interface; at least one processor connected to the above at least one bus; and at least one memory connected to the above at least one bus. The above memory stores a computer program, and the above processor is configured to implement the data processing method of the distributed computing engine described in any one of the above when executing the above computer program.

[0014] In a fourth aspect, the present application further provides a computer storage medium storing computer-executable instructions for executing the data processing method of the distributed computing engine described in any one of the above in the present application.

[0015] The above technical solution provided by the embodiments of the present application has the following advantages compared with the prior art: In the solution provided by the embodiments of the present application, custom functions can be placed in a custom function server, and the call interface of the custom function server can be declared in the distributed computing engine. In this way, when a structured query statement including a custom function is received, the function of the custom function server can be used to process the part of the custom function in the structured query statement through the call interface of the distributed computing engine, improving the processing efficiency of the structured query statement including the custom function. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention and, together with the specification, are used to explain the principles of the present invention.

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] One or more embodiments are exemplarily illustrated by the pictures in the corresponding accompanying drawings. These exemplary illustrations do not limit the embodiments. Elements with the same reference numerals in the drawings represent similar elements. Unless otherwise stated, the drawings in the figures do not constitute a proportional limitation.

[0019] Figure 1 It is a flowchart of a data processing method for a distributed computing engine provided by an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of a data processing system for a distributed computing engine provided by an embodiment of the present application;

[0021] Figure 3 It is a flowchart of another data processing method for a distributed computing engine provided by an embodiment of the present application;

[0022] Figure 4 It is a flowchart of yet another data processing method for a distributed computing engine provided by an embodiment of the present application;

[0023] Figure 5 It is a flowchart of yet another data processing method for a distributed computing engine provided by an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of another data processing system for a distributed computing engine provided by an embodiment of the present application;

[0025] Figure 7 It is a schematic diagram of the structure of a data processing device for a distributed computing engine provided by an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0028] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0029] To solve the technical problem of low processing efficiency of structured query statements including custom functions in the prior art, the present application provides a data processing method for a distributed computing engine, which can achieve the effect of improving the processing efficiency of structured query statements including custom functions.

[0030] Figure 1 It is a flowchart of a data processing method for a distributed computing engine provided by an embodiment of the present application. As Figure 1 shown, the data processing method for the above-mentioned distributed computing engine includes:

[0031] S102, when receiving a structured query statement from a client, parse the structured query statement to obtain a first task and a target task, where the structured query statement includes a custom function, the target task is the task corresponding to the custom function, and the first task is the task corresponding to the statement except the custom function;

[0032] S104, call the interface of the target custom function server to enable the target custom function server to execute the target task;

[0033] S106, return the target data generated by the target custom function server and the first data generated by the working node of the distributed computing engine to the client as the execution result of the structured query statement, where the first data is the data obtained by the working node of the distributed computing engine executing the first task.

[0034] The structured query statement (Structured Query Language, SQL) in this embodiment is a statement initiated by the client. Through the SQL statement, data can be queried. The SQL statement can be obtained by the distributed computing engine. The distributed computing engine can be a Trino engine, which includes a Coordinator and working nodes (Workers). The Coordinator is responsible for scheduling the user's SQL statement to the Workers, and the Workers are responsible for executing the tasks corresponding to the SQL statement, processing the data, and after processing the data, the distributed computing engine returns the processing results of the working nodes to the client.

[0035] The above SQL statement includes a user-defined function. After the distributed computing engine obtains the SQL statement, it can parse the statement, parsing the part of the user-defined function and the part outside the user-defined function. For the part of the user-defined function, it is parsed into a target task, and for the part outside the user-defined function, it is parsed into a first task.

[0036] In this embodiment, for the first task, the Trino engine can schedule the first task to a Worker for processing. For the target task, the interface of the user-defined function server can be called, and the user-defined function server is used to execute the target task and obtain the processing result of the target task from the user-defined function server.

[0037] Figure 2 It is a system schematic diagram of this embodiment. The system includes a distributed computing engine Trino and a user-defined function (UDF) server UDF Server. Among them, Trino loads the user-defined function interface, registers the user-defined function, executes the user-defined function, calls the user-defined function server to execute the task, and receives the result returned by the user-defined function server.

[0038] The solution provided by the embodiment of this application can place the user-defined function in the user-defined function server and declare the call interface of the user-defined function server in the distributed computing engine. In this way, when receiving a structured query statement including a user-defined function, the function of the user-defined function server can be used to process the part of the user-defined function in the structured query statement through the call interface of the distributed computing engine, improving the processing efficiency of the structured query statement including the user-defined function.

[0039] As an optional example, as Figure 3 shown, calling the interface of the target user-defined function server to enable the target user-defined function server to execute the target task includes:

[0040] S302, determining the function type of the user-defined function;

[0041] S304, determining the target user-defined function server from multiple user-defined function servers through the correspondence between the function type and the user-defined function server;

[0042] S306, calling the interface of the target user-defined function server to execute the target task.

[0043] In this embodiment, there can be multiple custom function servers, and the functions of the custom functions provided by different servers can be different. Different types of custom functions can be placed on different servers according to the categories of the custom functions, or multiple custom functions can be placed on one server, and then the corresponding relationship between the custom functions and the server can be recorded. For example, if the strategy of storing different types of custom functions on different servers is adopted, then after obtaining the target task, according to the function type of the custom function indicated by the target task, the corresponding target custom function server can be determined, and the target task can be sent to the target custom function server. If the method of recording the corresponding relationship between the custom function and the server is adopted, then the corresponding relationship between the custom function and the server can be queried through the function identifier of the custom function corresponding to the target task, so as to determine the target custom function server.

[0044] As an optional example, as Figure 4 shown, calling the interface of the target custom function server to execute the target task includes:

[0045] S402, when there is one target custom function server, call the target custom function server to execute the target task;

[0046] S404, when there are multiple target custom function servers, select an idle target custom function server to execute the target task.

[0047] In this embodiment, when determining the target custom function server, if there is one determined target custom function server, then the interface of the target custom function server can be called to execute the target task. If there are multiple determined target custom function servers, there are multiple corresponding processing methods. One of them is to randomly select an idle target custom function server from multiple target custom function servers to process the target task. If all target custom function servers are not idle, the target task can be stored in a queue until one target custom function server becomes idle. Another method is to select the server with the most central processing unit computing resources or the lowest load from multiple target custom function servers to process the target task.

[0048] As an optional example, before determining the target custom function server from multiple custom function servers through the corresponding relationship between the function type and the custom function server, the above method further includes: receiving a custom function set from the custom function server, where the custom function set records the custom functions included in the custom function server; identifying the types of the custom functions in the custom function set to establish the corresponding relationship between the function type and the custom function server.

[0049] In this embodiment, a custom function server can store a type of custom function. The custom functions can be classified in advance, and then, for each type of custom function, a corresponding custom function server is prepared. Each custom function server stores a type of custom function. For a custom function server, the sizes of the central processing unit resources, memory resources, and disk resources of the custom function server used can be determined according to the complexity of the custom function. The complexity of the custom function can be calculated based on the number of calculation steps performed during the execution of the custom function. The more steps, the higher the complexity.

[0050] As an optional example, as Figure 5 shown, calling the interface of the target custom function server to enable the target custom function server to execute the target task includes:

[0051] S502, searching for the target data corresponding to the target task from the local cache;

[0052] S504, in the case where the target data is found, prohibiting the calling of the interface of the target custom function server;

[0053] S506, in the case where the target data is not found, calling the interface of the target custom function server.

[0054] In this embodiment, for the custom function server, after the target task is executed to obtain the target data, the distributed computing engine can obtain the target data returned by the custom function server. Then, in addition to sending the target data to the client, the distributed computing engine can also store the target data in the cache. The target data in the cache is deleted regularly or according to the remaining space size of the cache, so as to keep the remaining space size of the cache greater than a critical value. When the distributed computing engine receives an SQL statement and parses it to obtain the target task, it first searches the cache to see if there is target data for the target task. If there is target data, it directly obtains the target data from the cache instead of handing the target task to the custom function server for processing, thus improving the efficiency of obtaining the target data. If the target data is not in the cache, the target task is then handed to the custom function server for processing to obtain the target data.

[0055] As an optional example, in the case of receiving a structured query statement from the client, parsing the structured query statement to obtain the target task includes: parsing the structured query statement to obtain the client identifier of the client; inserting the client identifier into the target task. When the target custom function server executes the target task, it verifies the access permission of the client through the client identifier and determines whether to execute the target task according to the access permission.

[0056] In this embodiment, different clients may correspond to different access permissions. The different access permissions are to access different custom functions or different custom function servers. For example, taking the custom function server as an example, when receiving a structured query statement from a client, the client identifier of the client is inserted into the target task, and then the target task is sent to the target custom function server. The target custom function server determines whether the client has the permission to use the target custom function server through the client identifier. If the client has no permission, an error message is returned.

[0057] As an alternative example, the method further includes: each custom function server monitors the size of its total storage space and the size of its available storage space; when the ratio of the size of the available storage space to the size of the total storage space of the current custom function server is less than a predetermined threshold, the current custom function server is expanded.

[0058] In this embodiment, the total storage space of the custom function server can be expanded and contracted. Specifically, it can be judged according to the size of the total storage space and the size of the available storage space. If the ratio of the size of the available storage space to the size of the total storage space is less than a predetermined threshold, it means that the custom function server needs to be expanded, and the available storage space will increase after expansion. The specific expansion value can be determined according to the size of the total storage space of the custom function, and can be a certain proportion of the total storage space.

[0059] Figure 6 is the system schematic diagram of this embodiment. The custom function interface server of this embodiment can be deployed on the cloud hosts of different cloud providers or on local physical machines. When Trino queries data, according to the registered address, it uses the custom high-speed protocol KingProtocolBuffer protocol to send a function call calculation request to the specified server. The KingProtocolBuffer protocol adds a caching logic internally, which can cache the values calculated by the UDF server. If the value requested by Trino has been calculated, the protocol directly returns the request of Trino without sending it to the UDF server, but directly obtains the data from the cache. The cloud host on which the custom function interface server is deployed should provide hardware computing power, hardware storage capacity, network communication bandwidth, and the computing cost per unit time for the custom function interface server to choose for deployment. For different tenants with confidentiality and security requirements, during deployment, it is deployed in a restricted environment and the permissions are verified during access. The deployed custom function server can expand or contract the cloud host according to different loads.

[0060] Figure 7 is the structural schematic diagram of a data processing device of a distributed computing engine provided by an embodiment of the present application. AsFigure 7 As shown in the figure, the data processing device of the above distributed computing engine includes:

[0061] A parsing module 702, configured to parse a structured query statement received from a client to obtain a first task and a target task. The structured query statement includes a user-defined function, the target task is a task corresponding to the user-defined function, and the first task is a task corresponding to the statement other than the user-defined function;

[0062] An invocation module 704, configured to invoke an interface of a target user-defined function server to enable the target user-defined function server to execute the target task;

[0063] A sending module 706, configured to return the target data generated by the target user-defined function server and the first data generated by a working node of the distributed computing engine to the client as the execution result of the structured query statement, where the first data is the data obtained by the working node of the distributed computing engine executing the first task.

[0064] The structured query statement (Structured Query Language, SQL) in this embodiment is a statement initiated by the client. Through the SQL statement, data can be queried. The SQL statement can be obtained by the distributed computing engine, and the distributed computing engine can be a Trino engine, which includes a coordinator and working nodes. The coordinator is responsible for scheduling the user's SQL statement to the working nodes, and the working nodes are responsible for executing the tasks corresponding to the SQL statement and processing the data. After processing the data, the distributed computing engine returns the processing results of the working nodes to the client.

[0065] The above SQL statement includes a user-defined function. After the distributed computing engine obtains the SQL statement, it can parse the statement, parsing the user-defined function part and the part other than the user-defined function. For the user-defined function part, it is parsed into a target task, and for the part other than the user-defined function, it is parsed into a first task.

[0066] In this embodiment, for the first task, the Trino engine can schedule the first task to the working nodes for processing. For the target task, the interface of the user-defined function server can be invoked, and the user-defined function server is responsible for executing the target task and obtaining the processing result of the target task from the user-defined function server.

[0067] Figure 2It is a schematic diagram of the system in this embodiment. The system includes a distributed computing engine Trino and a User Define Function (UDF) server UDF Server. Trino in it loads the UDF interface, registers the UDF, executes the UDF, calls the UDF server to execute tasks, and receives the results returned by the UDF server.

[0068] The solution provided in the embodiment of this application can place the UDF in the UDF server and declare the call interface of the UDF server in the distributed computing engine. In this way, when receiving a structured query statement including a UDF, the function of the UDF server can be used to process the UDF part in the structured query statement through the call interface of the distributed computing engine, improving the processing efficiency of the structured query statement including the UDF.

[0069] For other examples of this embodiment, please refer to the above examples and will not be elaborated here.

[0070] As Figure 8 shown, the embodiment of this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. Among them, the processor 111, the communication interface 112, and the memory 113 complete mutual communication through the communication bus 114.

[0071] The memory 113 is used to store computer programs.

[0072] In an embodiment of this application, when the processor 111 executes the program stored on the memory 113, it implements the data processing method of the distributed computing engine provided in any of the foregoing method embodiments.

[0073] The embodiment of this application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the data processing method of the distributed computing engine provided in any of the foregoing method embodiments.

[0074] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0076] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "comprising", "including", "containing", and "having" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or their combinations. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the particular order described or illustrated, unless the order of performance is explicitly stated. It should also be understood that additional or alternative steps may be used.

[0077] The above description is only the specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data processing method for a distributed computing engine, characterized in that, it includes: When receiving a structured query statement from a client, parsing the structured query statement to obtain a first task and a target task, where the structured query statement includes a custom function, the target task is the task corresponding to the custom function, and the first task is the task corresponding to the statement except the custom function; Invoking the interface of the target custom function server to enable the target custom function server to execute the target task; Returning the target data generated by the target custom function server and the first data generated by the working nodes of the distributed computing engine as the execution result of the structured query statement to the client, where the first data is the data obtained by the working nodes of the distributed computing engine executing the first task.

2. The method according to claim 1, characterized in that, The invoking the interface of the target custom function server to enable the target custom function server to execute the target task includes: Determining the function type of the custom function; Determining the target custom function server from multiple custom function servers through the correspondence between the function type and the custom function server; Invoking the interface of the target custom function server to execute the target task.

3. The method according to claim 2, characterized in that, The invoking the interface of the target custom function server to execute the target task includes: When the target custom function server is one, invoking the target custom function server to execute the target task; When there are multiple target custom function servers, selecting an idle target custom function server to execute the target task.

4. The method according to claim 2, characterized in that, Before determining the target custom function server from multiple custom function servers through the correspondence between the function type and the custom function server, the method further includes: Receiving a custom function set from the custom function server, where the custom function set records the custom functions included in the custom function server; Identifying the types of the custom functions in the custom function set to establish the correspondence between the function type and the custom function server.

5. The method according to claim 1, characterized in that, The invoking the interface of the target custom function server to enable the target custom function server to execute the target task includes: Searching for the target data corresponding to the target task in the local cache; When the target data is found, prohibiting the invocation of the interface of the target custom function server; When the target data is not found, invoking the interface of the target custom function server.

6. The method according to claim 1, characterized in that, The parsing the structured query statement to obtain the target task when receiving a structured query statement from a client includes: Parse the structured query statement to obtain the client identifier of the client; Insert the client identifier into the target task. When the target custom function server executes the target task, it verifies the access permission of the client through the client identifier and determines whether to execute the target task according to the access permission.

7. The method according to claim 1, wherein, the method further includes: Each custom function server monitors the size of its total storage space and the size of its available storage space; When the ratio of the size of the available storage space to the size of the total storage space of the current custom function server is less than a predetermined threshold, the current custom function server is expanded.

8. A data processing device for a distributed computing engine, wherein, it includes: A parsing module, configured to parse the structured query statement received from the client to obtain a first task and a target task, where the structured query statement includes a custom function, the target task is a task corresponding to the custom function, and the first task is a task corresponding to the statement other than the custom function; A calling module, configured to call the interface of the target custom function server to cause the target custom function server to execute the target task; A sending module, configured to return the target data generated by the target custom function server and the first data generated by the working node of the distributed computing engine as the execution result of the structured query statement to the client, where the first data is the data obtained by the working node of the distributed computing engine executing the first task.

9. An electronic device, wherein, it includes: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing computer-executable instructions for executing the method described in any one of claims 1 to 7 of the present application.

Images