Data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device, a storage medium and electronic equipment. The method comprises the steps that a data processing request is received, and the data processing request at least comprises to-be-processed target data and a target processing mode of the target data; according to a target processing mode in the data processing request, generating a structured query statement used for calling a first custom function; and calling a first custom function from a target interface of the target database according to the structured query statement, and processing the target data according to the first custom function to obtain a processing result. According to the data processing method and device, the technical problem that in the prior art, a user needs to compile, package, add resources, create functions and the like to process the data through the self-defined functions, and consequently the data processing efficiency is low is solved.

Description

Technical Field

[0001] This application relates to the technical field of data processing. Specifically, it relates to a data processing method, device, storage medium, and electronic device. Background Art

[0002] The user-defined function UDF (User Defined Function) is the most core capability in relational databases, especially distributed large database products, and is the most widely used sub-function in big data products. The user-defined function effectively extends the limited built-in function capabilities of the database and supports users to customize various special task logics and requirements to implement various complex task logics. The built-in functions and user-defined functions in SQL (Structured Query Language) statements are one of the most commonly used tools in SQL usage and are also the sub-functions with the highest usage rate in various big database products on the cloud. From the pipelined table functions in oracle, the table-valued functions and related stored procedures in ms-sql to the udtf and udaf on maxcompute, hologres, and various open-source distributed big data platforms, with the help of the user-defined function UDF, the SQL engine can implement complex demand task logics and meet the complex needs of users.

[0003] There are various syntax definitions, designs, and architecture implementations of the user-defined function UDF. Most databases, especially traditional databases, support writing and encapsulating user-defined functions using SQL language, also known as PL / SQL (Procedure Language / SQL). However, the current user-defined function UDF requires users to perform operations such as compilation (Java) packaging, adding resources, and creating functions before it can be used, and the process is relatively cumbersome. Additionally, it is necessary to establish a project locally, compile, package, release and go online, and create a user-defined function based on the package. When the task logic is updated, the whole process needs to be repeated. Moreover, the encapsulated program is a black box, which is not convenient for viewing and modification. If SQL contains UDF, its implementation logic cannot be directly viewed, and even the source code of some jar packages cannot be found, which brings certain inconveniences to code maintenance.

[0004] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of this application provide a data processing method, device, storage medium, and electronic device to at least solve the technical problem in the prior art that data processing through user-defined functions can only be performed after operations such as user compilation and packaging, adding resources, and creating functions, resulting in relatively low data processing efficiency.

[0006] According to one aspect of the embodiments of the present application, a data processing method is provided, including: receiving a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions for target code; calling the first custom function from a target interface of a target database according to the structured query statement, and processing the target data according to the first custom function to obtain a processing result.

[0007] Further, before generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, the method further includes: generating target code according to the general programming language and the target processing method; generating the first custom function according to the custom function template and the target code.

[0008] Further, generating the first custom function according to the custom function template and the target code includes: adding the target code to a target position in the custom function template to obtain an initial custom function; performing setting processing on target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage class keywords and update class keywords; performing analysis processing on the processed initial custom function to obtain the first custom function.

[0009] Further, performing analysis processing on the processed initial custom function to obtain the first custom function includes: generating an abstract syntax tree according to the processed initial custom function; performing error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result; performing compilation analysis according to the processed initial custom function to obtain a second analysis result; correcting the processed initial custom function according to the first analysis result and the second analysis result to obtain the first custom function.

[0010] Further, generating an abstract syntax tree according to the processed initial custom function includes: splitting the processed initial custom function to obtain multiple strings; filtering the multiple strings to obtain processed strings; generating the abstract syntax tree according to the processed strings.

[0011] Further, error analysis is performed on the code in the first custom function according to the abstract syntax tree, and the first analysis result is obtained, including: according to the abstract syntax tree, analyzing the data processing process of the code in the first custom function in a real execution environment, and obtaining a first analysis sub-result; according to the abstract syntax tree, performing a physical analysis on the data processing process of the code in the first custom function, and obtaining a second analysis sub-result; and obtaining the first analysis result according to the first analysis sub-result and the second analysis sub-result.

[0012] Further, compilation analysis is performed according to the processed initial custom function, and the second analysis result is obtained, including: according to the processed initial custom function, obtaining the target code; in a real execution environment, performing compilation on the target code to obtain an execution result, where the real execution environment is the real execution environment of the first custom function; and performing error analysis according to the execution result to obtain the second analysis result.

[0013] Further, after obtaining the first custom function according to the custom function template and the target code, the method further includes: persistently storing the first custom function in the target database; and performing an extension process on the target interface of the target database to call the first custom function through the extended target interface.

[0014] According to another aspect of the embodiments of the present application, a data processing method is further provided, including: obtaining a data processing request uploaded by a client, where the data processing request at least includes: target data to be processed and a target processing method for the target data; generating a structured query statement for calling a first custom function in a cloud server according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions for the target code, calling the first custom function from the target interface of the target database according to the structured query statement, and processing the target data according to the first custom function to obtain a processing result; and returning the processing result to the client.

[0015] According to another aspect of the embodiments of the present application, a data processing device is further provided, including: a receiving unit, configured to receive a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; a first generating unit, configured to generate a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions for target code; a calling unit, configured to call the first custom function from a target interface of a target database according to the structured query statement, and process the target data according to the first custom function to obtain a processing result.

[0016] Further, the device further includes: a second generating unit, configured to generate target code according to the general programming language and the target processing method before generating a structured query statement for calling a first custom function according to the target processing method in the data processing request; a third generating unit, configured to generate the first custom function according to the custom function template and the target code.

[0017] Further, the third generating unit includes: an adding subunit, configured to add the target code to a target position in the custom function template to obtain an initial custom function; a setting subunit, configured to perform setting processing on target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage class keywords and update class keywords; an analyzing subunit, configured to perform analyzing processing on the processed initial custom function to obtain the first custom function.

[0018] Further, the analyzing subunit includes: a generating module, configured to generate an abstract syntax tree according to the processed initial custom function; a first analyzing module, configured to perform error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result; a second analyzing module, configured to perform compilation analysis according to the processed initial custom function to obtain a second analysis result; a correcting module, configured to correct the processed initial custom function according to the first analysis result and the second analysis result to obtain the first custom function.

[0019] Further, the generating module includes: a splitting sub-module, configured to perform splitting processing on the processed initial custom function to obtain a plurality of strings; a filtering sub-module, configured to perform filtering processing on the plurality of strings to obtain processed strings; a generating sub-module, configured to generate the abstract syntax tree according to the processed strings.

[0020] Further, the first analysis module includes: a first analysis sub-module, configured to analyze the data processing process of the code in the first custom function based on the abstract syntax tree in a real execution environment-free manner to obtain a first analysis sub-result; a second analysis sub-module, configured to physically analyze the data processing process of the code in the first custom function based on the abstract syntax tree to obtain a second analysis sub-result; and a determination sub-module, configured to obtain the first analysis result based on the first analysis sub-result and the second analysis sub-result.

[0021] Further, the second analysis module includes: an acquisition sub-module, configured to obtain the target code based on the processed initial custom function; a compilation sub-module, configured to perform compilation on the target code in a real execution environment to obtain an execution result, where the real execution environment is the real execution environment of the first custom function; and a third analysis sub-module, configured to perform error analysis based on the execution result to obtain the second analysis result.

[0022] Further, the apparatus further includes: a storage unit, configured to persistently store the first custom function in the target database after obtaining the first custom function based on the custom function template and the target code; and an extension unit, configured to perform extension processing on a target interface of the target database to call the first custom function through the extended target interface.

[0023] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a program, where when the program runs, it controls a device where the storage medium is located to execute the data processing method described in any one of the above.

[0024] According to another aspect of the embodiments of the present invention, there is also provided an electronic device including a memory storing an executable program; and a processor configured to run the program, where when the program runs, it executes the data processing method described in any one of the above.

[0025] In the embodiment of the present application, a data processing request is received. The data processing request at least includes: target data to be processed and a target processing method for the target data. According to the target processing method in the data processing request, a structured query statement for calling a first custom function is generated. The first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide the execution conditions of the target code. The first custom function is called from the target interface of the target database according to the structured query statement, and the target data is processed according to the first custom function to obtain a processing result, which solves the technical problem in the prior art that the user needs to perform operations such as compilation and packaging, resource addition, and function creation before the data can be processed through the custom function, resulting in relatively low data processing efficiency. In this solution, when it is necessary to process the target data, a structured query statement for calling the first custom function can be directly generated according to the current target processing method, and finally the first custom function is called from the target interface of the target database by using the structured query statement to complete the processing of the target data, avoiding the need for the user to perform operations such as compilation and packaging, resource addition, and function creation before the data can be processed through the custom function. Moreover, the custom function in this solution is obtained based on a general programming language, which improves the writing efficiency compared with writing and encapsulating custom functions through SQL language in the prior art. And the custom function in this solution supports being created once and used multiple times, avoiding the need for the user to perform operations such as establishing a project locally, compiling, packaging, and releasing online, thereby achieving the effect of improving the data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0027] Figure 1 is a schematic diagram of a computer terminal provided in Embodiment 1 of the present application;

[0028] Figure 2 is a flowchart of a data processing method provided in Embodiment 1 of the present invention;

[0029] Figure 3 is a schematic diagram of a custom function provided in Embodiment 1 of the present invention;

[0030] Figure 4 is a flowchart of writing a custom function in the prior art;

[0031] Figure 5 is a flowchart of writing a custom function provided in Embodiment 1 of the present invention;

[0032] Figure 6 It is a flowchart of the data processing method provided in Embodiment 2 of the present invention;

[0033] Figure 7 It is a schematic diagram of the data processing device provided in Embodiment 3 of the present invention;

[0034] Figure 8 It is a schematic diagram of the computer terminal provided in Embodiment 4 of the present invention. Detailed implementation manners

[0035] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0037] First, some nouns or terms that appear during the description of the embodiments of the present application are applicable to the following explanations:

[0038] Distributed SQL: Structured Query Language, a standard language for database queries; the SQL language corresponds to the relational data computing model. Distributed SQL means that a distributed system supports the scheduling and transformation of relational data computing jobs submitted in SQL language to execute in a distributed parallel environment. The execution process of a distributed SQL job is first to parse the SQL into an abstract syntax tree and convert it into a logical plan. The logical plan is a DAG (directed acyclic graph) composed of relational operators, and then convert the logical plan into a physical execution plan and schedule it for distributed operation on a distributed computing platform.

[0039] User-Defined Function (UDF): In the narrow sense, UDF refers to User Defined Scalar Function. In the broad sense, user-defined function UDF includes User Defined Scalar Function, User Defined Table-Generating Function (UDTF), and User Defined Aggregation Function (UDAF). In this application, the user-defined function UDF refers to the user-defined function UDF in the broad sense, including various user-defined functions. The user-defined function UDF (User Defined Function) in the database can support various special task logics and requirements of users for custom extensions. Among them, UDF (User Defined Scalar Function) refers to the user-defined scalar function, usually called UDF. Its input and output are in a one-to-one relationship, that is, reading a row of data and writing out a single output value; UDTF (User Defined Table-valued Function) is a user-defined table-valued function used to solve the scenario where a single function call outputs multiple rows of data and is also the only user-defined function that can return multiple fields; UDAF (User Defined Aggregation Function) is a user-defined aggregation function, and its input and output are in a many-to-one relationship, that is, aggregating multiple input records into a single output value, which can be used in combination with the Group By statement in SQL.

[0040] General - purpose programming language: A programming language is a formal language used to define the execution flow of computer instructions. Each programming language consists of a complete set of vocabulary and grammar specifications. These specifications usually include data types and data structures, instruction types and instruction control, call mechanisms and library functions, as well as unwritten rules (such as progressive writing, variable naming, etc.). There are various classification methods for programming languages. Most programming languages are algorithm - description languages, such as C / C++, Java, etc., and some are data - description languages, such as markup languages like HTML. According to the difficulty of programming techniques, they can be divided into low - level languages (machine language, assembly language) and high - level languages; according to the programming language design style, they can be divided into imperative languages (procedural languages), structured languages, object - oriented languages, functional languages, scripting languages, etc.; according to the language application field, they can be divided into general - purpose programming languages and special - purpose programming languages; according to the program execution method, they can be divided into interpreted languages (such as JavaScript, Python, Perl, R, etc.), compiled languages (such as C / C++), and compiled + interpreted languages (such as Java, PHP, etc.). The general - purpose programming languages in this application are classified according to the wide - spread degree of use, including Java, Python, etc., which are the most widely used and powerful.

[0041] User - free compilation: Compilation refers to the process of using a compiler to generate a target program from a source program written in a source language and the action of generating a target program using a compiler. For example, the compilation of a programming language is the translation process of translating a source program written in a high - level computer language such as C++, Java, C# into a machine language that can be executed on a computer, converting the high - level language into a binary language that the computer can recognize. The computer only recognizes 1 and 0, and the compiler converts the language familiar to people into binary. In the existing custom function UDF, the source code of the custom function UDF needs to be encapsulated as an executable code and a jar package and submitted and published to the computing platform before compilation processing. However, in this application, the custom function UDF does not require user compilation, that is, user - free compilation.

[0042] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards in the relevant region, and a corresponding operation entry is provided for the user to choose to authorize or refuse.

[0043] Embodiment 1

[0044] According to an embodiment of the present application, a data processing method is further provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0045] The method embodiment provided by the first embodiment of the present application can be executed in a mobile terminal, a computer terminal, or a similar computing device. Figure 1 The following shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing the data processing method. As Figure 1 shown, the computer terminal (or mobile device) 10 may include a set of processors 102 (the set of processors 102 may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA, and the set of processors 102 may include a set of processors, Figure 1 where 102a, 102b,..., 102n are used to represent), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may further include: a display, an input / output interface (I / O interface), a universal serial bus (USB, Universal Serial Bus) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply, and / or a camera. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown, or have a different configuration from Figure 1 shown.

[0046] It should be noted that the above one or more processors 102 and / or other data processing circuits can generally be referred to as "data processing circuits" in this document. The data processing circuit can be embodied in whole or in part as software, hardware, firmware, or any arbitrary combination. In addition, the data processing circuit can be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit is a kind of processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage devices corresponding to the data processing method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned data processing method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include but are not limited to the Internet, enterprise intranets, local area networks, mobile communication networks, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] The display can be, for example, a touch-screen liquid crystal display (LCD), and the liquid crystal display enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

[0050] Under the above operating environment, the present application provides a data processing method as Figure 2 shown. Figure 2 is a flowchart of the data processing method according to Embodiment 1 of the present application, including:

[0051] Step S201, receiving a data processing request, where the data processing request at least includes: target data to be processed and a target processing manner for the target data.

[0052] Optionally, obtain a data processing request, and the data processing request should at least include target data to be processed and what to do with the target data (that is, the above-mentioned target processing manner). For example, the target processing manner can be filtering the data, or performing a summation operation on the data, etc.

[0053] Step S202: Generate a structured query statement for calling a first custom function according to the target processing method in the data processing request. The first custom function is obtained based on a custom function template and a general-purpose programming language. The custom function template is used to provide the execution conditions for the target code.

[0054] Optionally, after obtaining the above-mentioned target processing method, determine which custom function to use for data processing according to the target processing method, that is, generate a structured query statement for calling the first custom function.

[0055] It should be noted that the first custom function is obtained based on a custom function template and a general-purpose programming language. The general-purpose programming language can be programming languages such as Java and Python. It should be noted that the above-mentioned custom function template is used to provide the execution conditions for the target code, that is, to enable the target code to be executed through the custom function model. The above-mentioned custom function template supports using general-purpose programming languages such as Java and Python to write task logic in the function body, which can effectively improve the writing efficiency of custom functions and reduce the code maintenance cost.

[0056] Step S203: Call the first custom function from the target interface of the target database according to the structured query statement, and process the target data according to the first custom function to obtain a processing result.

[0057] Optionally, after obtaining the above-mentioned structured query statement, execute the structured query statement, call the first custom function from the target interface of the target database, and obtain the target data to be processed. Finally, process the target data according to the first custom function to obtain a processing result.

[0058] In summary, when data processing of target data is required, a structured query statement for calling a first custom function can be directly generated according to the current target processing method. Finally, the first custom function is called from the target interface of the target database using the structured query statement to complete the processing of the target data, avoiding the need for users to perform operations such as compilation and packaging, adding resources, and creating functions before processing data through custom functions. Moreover, the custom function in this solution is obtained based on a general-purpose programming language. Compared with writing and encapsulating custom functions using SQL language in the prior art, the writing efficiency is improved. In addition, the custom function in this solution supports being created once and used multiple times, avoiding the need for users to perform operations such as establishing a project locally, compiling, packaging, and releasing online, thereby achieving the effect of improving the data processing efficiency.

[0059] In order to be able to directly call the first custom function to process target data, in the data processing method provided in the first embodiment of this application, before generating a structured query statement for calling the first custom function according to the target processing method in the data processing request, the method further includes: generating target code according to a general programming language and the target processing method; generating the first custom function according to a custom function template and the target code.

[0060] Optionally, in the data processing method provided in this application, the following steps are used to obtain the above-mentioned first custom function: according to the target processing method for target data, use a programming language to write task logic, that is, generate the above-mentioned target code, and then obtain a custom function template, and generate the above-mentioned first custom function through the custom function template and the target code.

[0061] It should be noted that the custom function template supports using general programming languages such as Java and Python to write task logic as the function body, which can effectively improve the writing efficiency of custom functions and reduce the code maintenance cost.

[0062] The custom function template also includes target keywords. For example, the embedded storage class keyword can be used to persistently store the custom function obtained based on the custom function template by setting this keyword.

[0063] In order to implement the feasibility of the first custom function, in the data processing method provided in the first embodiment of this application, generating the first custom function according to a custom function template and target code includes: adding the target code to the target position in the custom function template to obtain an initial custom function; performing setting processing on the target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage class keywords and update class keywords; performing analysis processing on the processed initial custom function to obtain the first custom function.

[0064] Optionally, the two characters #code and #end code included in the custom function template are used to add the target code obtained based on the task processing logic between these two, that is, adding the target code to the target position of the custom function template as described above. After obtaining the above-mentioned initial custom function, set the target keywords. For example, embedded function is a required keyword (that is, the above-mentioned storage class keyword), indicating that the task code written in the programming language needs to be persistently stored and can be used multiple times after successful creation. The replace keyword (that is, the above-mentioned update class keyword), for an existing function, replace means that saving takes effect immediately.

[0065] In an optional embodiment, the first custom function obtained based on the above custom function template is as follows Figure 3 shown. "embedded function" is a required keyword, indicating that the task code written in a programming language needs to be persistently stored and can be used multiple times after successful creation. The "replace" keyword is optional. "create or replace function" means that it supports updating the task processing code content of an existing function. For an existing function, "replace" means that saving it takes effect immediately. Inside the "#code" and "#end code" after the "using" keyword, the general programming languages Java and Python can be used to write task processing. The language type is configured after "lang". For example, Figure 3 in " Figure 3 ", 'lang' = 'JAVA' represents that the programming language is Java.

[0066] After obtaining the processed initial custom function as described above, perform analysis processing on the processed initial custom function, such as lexical analysis, syntax analysis, semantic analysis, type checking, constant folding, and other processings, to obtain the above first custom function.

[0067] Through the above steps, the user does not need to manage software resource packages, nor does the user need to handle various complex processes such as compilation, packaging, and uploading, thereby achieving the improvement of the efficiency of data processing.

[0068] To improve the accuracy and reusability of the first custom function, in the data processing method provided in the first embodiment of this application, the analysis processing of the processed initial custom function to obtain the first custom function includes: generating an abstract syntax tree based on the processed initial custom function; performing error analysis on the code in the first custom function based on the abstract syntax tree to obtain a first analysis result; performing compilation analysis based on the processed initial custom function to obtain a second analysis result; and correcting the processed initial custom function based on the first analysis result and the second analysis result to obtain the first custom function.

[0069] Optionally, after obtaining the processed initial custom function as described above, parse the processed initial custom function to obtain the above abstract syntax tree. The abstract syntax tree (AST) is a tree-shaped structure representation of the user input statement. Each node on the tree is a word, and the structure of the tree reflects the syntax. The abstract syntax tree is constructed during the syntax analysis process. When the syntax analysis ends, the syntax analyzer will output an abstract syntax tree, and the user input and the structural content of the abstract syntax tree are in one-to-one correspondence.

[0070] After obtaining the above abstract syntax tree, error analysis is performed on the code in the first custom function through the abstract syntax tree. For example, the feasibility and rationality of the code are analyzed to obtain the above first analysis result.

[0071] After error analysis, it is necessary to perform compilation analysis on the processed initial custom function, which can reduce various incompatibility problems in the real production environment after local compilation in the prior art. And by performing compilation analysis on the custom function, problems of type mismatch can be detected early, without waiting until execution to discover this problem, thus discovering problems in advance and reducing usage costs.

[0072] Finally, the processed initial custom function is corrected according to the first analysis result and the second analysis result to obtain the final first custom function. For example, if the first analysis result indicates that there is a mismatch between the actual type and the declared type of a variable or parameter, or the second analysis result indicates that there is a compilation error, then the code in the processed initial custom function is corrected according to these problems to obtain the above first custom function.

[0073] Through the above error analysis and compilation analysis, various incompatibility problems in the subsequent real production environment can be effectively avoided, and the long-chain manual time-consuming operations of traditional user-defined functions (UDFs) are automated inside the database engine. Users do not need to manage software resource packages and do not need to handle various complex processes such as compilation, packaging, and uploading, which improves the efficiency of writing custom functions.

[0074] In an optional embodiment, in the data processing method provided in the first embodiment of this application, generating an abstract syntax tree based on the processed initial custom function includes: splitting the processed initial custom function to obtain multiple strings; filtering the multiple strings to obtain processed strings; and generating an abstract syntax tree based on the processed strings.

[0075] Optionally, the processed initial custom function is split into character segments Token including keyword recognition, that is, the above multiple strings are obtained. To improve the simplicity of subsequent construction of the abstract syntax tree, the multiple strings can be filtered to obtain processed strings.

[0076] After obtaining the processed strings, the Token stream can be parsed into an abstract syntax tree using a top-down or bottom-up algorithm.

[0077] In order to improve the accuracy of error analysis for code, in the data processing method provided in the first embodiment of this application, the code in the first custom function is subjected to error analysis based on the abstract syntax tree, and the first analysis result is obtained, including: based on the abstract syntax tree, analyzing the data processing process of the code in the first custom function in a real execution environment-free manner to obtain the first sub-analysis result; based on the abstract syntax tree, performing physical analysis on the data processing process of the code in the first custom function to obtain the second sub-analysis result; and obtaining the first analysis result based on the first sub-analysis result and the second sub-analysis result.

[0078] Optionally, according to the abstract syntax tree, logical analysis is performed on the data processing process of the code in the first custom function. It should be noted that logical analysis is to analyze the data processing process of the code in the first custom function in a real execution environment-free manner. The logical analysis process is basically a pure algebraic analysis process and has nothing to do with the underlying distributed environment. The overall logical analysis process is to analyze the input, complete process, and operations of the SQL statement, and during this period, call the task processing code of the custom function to execute and process the data.

[0079] After the logical analysis is completed, physical analysis is performed on the data processing process of the code in the first custom function. Physical analysis is closely related to the underlying execution environment. For example, during physical analysis, it is necessary to determine how to partition data, sort, read the amount of data, and start how many processes to execute tasks during Shuffle, etc.

[0080] It should be noted that during the logical analysis and physical analysis processes, type checking can also be performed, such as determining whether the actual type of a variable or parameter matches the declared type, to avoid problems when using the custom function later.

[0081] Through the above steps, the accuracy of the code of the first custom function is ensured.

[0082] In order to improve the accuracy of compilation analysis for the processed initial custom function, in the data processing method provided in the first embodiment of this application, compilation analysis is performed based on the processed initial custom function, and the second analysis result is obtained, including: obtaining the target code based on the processed initial custom function; in the real execution environment, performing compilation on the target code to obtain the execution result, where the real execution environment is the real execution environment of the first custom function; and performing error analysis based on the execution result to obtain the second analysis result.

[0083] Optionally, when performing compilation analysis on the processed initial custom function, obtain the target code based on the processed initial custom function, and then perform compilation checks for the persistent Embedded Function and compile and run it in the actual execution environment to obtain the second analysis result mentioned above.

[0084] It should be noted that for the target code written in Java, which accounts for the vast majority during compilation processing, Janino can be used for automatic compilation, packaging, and type checking. Janino is an extremely small and fast open-source Java compiler that can dynamically compile Java source code into Java bytecode at runtime and then execute the bytecode. The main features of Janino include dynamic compilation at runtime, lightweight and easy integration, support for dynamic class loading, etc. Janino can dynamically generate and load classes without pre-compiling Java source code, thus making Java programs more flexible and scalable. Janino can not only compile a set of Java source files into a set of bytecode class files like javac, but also compile Java expressions, code blocks, classes, and.java files in memory, load the bytecode, and directly execute it in the JVM.

[0085] By compiling the custom function in the actual execution environment, various incompatibility problems in the actual production environment after local compilation in the prior art can be effectively avoided.

[0086] In an optional embodiment, in the data processing method provided in the first embodiment of this application, after obtaining the first custom function based on the custom function template and the target code, the method further includes: persistently storing the first custom function in the target database; performing extension processing on the target interface of the target database to call the first custom function through the extended target interface.

[0087] Optionally, after obtaining the first custom function mentioned above, persistently store the first custom function and the relevant description information of the first custom function in the target database. In an optional embodiment, the above target database can be the Meta framework (meta-framework) module of the database platform. The meta-information Meta supports persistent embedding of Java / Python UDF and needs to define the corresponding storage structure. It supports creating a DDL Plan for Embedded function. The DDL plan is defined by SQL and is similar to the ordinary CreateFunction interface. At the same time, it extends the sdk interface of meta (i.e., the above target interface) so that sql can obtain the information of the Embedded function persisted in Meta and support displaying relevant information when describing the function.

[0088] In an optional embodiment, after the persistent Embedded Function is established, all established function lists can be viewed through list functions such as ls functions - p my_project, and the established EmbedFunc function can be deleted through drop Function such as drop Function myEmbedfuncName. Calling the Embed Func is the same as calling a general function, directly used in SQL, such as select my Embed func Name(col1, col2, col3) from t. When users modify and maintain it, they can directly execute the desc command on the platform to obtain the latest program source code and directly modify it, which can achieve the effect of improving the maintenance efficiency of UDFs.

[0089] The database persistent Embedded Function that supports general - purpose programming languages and does not require user compilation designed and constructed through this application allows users to simply write a program and save it to the platform for use when creating. There is no need, like the current industry's user - defined function, to establish an environment offline, write a program, compile and package, upload and update, define resources, and define functions based on resources.

[0090] The UDF designed in this application takes effect immediately after being checked, modified, and seen as the result. It is efficient in writing and using, and convenient in maintenance. When users modify and maintain it, they can directly execute the desc command on the platform to obtain the latest program source code and directly modify it, which improves the maintainability of UDFs and solves the problems existing in the current industry's UDFs, such as the need to independently save the source code and compilation package offline, separate management of the source code and execution package, difficulties in maintenance, and various problems such as source code loss or version mismatch when modifying some historical functions. As Figure 4 shown in the established process of the user - defined function in the prior art, it is necessary to manage resources and functions separately, and it is necessary to establish an environment offline, write a program, compile and package, upload and update, define resources, and define functions based on resources. The process of the UDF designed in this application is as Figure 5 shown. Users do not need to establish a project offline and store and maintain the task logic through a repository, nor do they need to import various underlying and application - layer libraries to establish the user - defined function. Users directly write a function containing the task logic, then register the function, and then upload the function to the database platform for storage, and perform compilation, packaging, and resource upload on the database platform. When users need to process data, they call the UDF function through SQL, and then complete the data - processing work on the database platform.

[0091] When the custom function provided by this application is created or updated, resolve is used for compilation verification during the creation process to detect problems in advance and reduce usage costs. After EmbedFunc is created, persistent storage is implemented to enable creation once and usage multiple times. It supports efficient execution with pre-check and persistent storage. When creating, it checks for program problems in the production environment during compilation verification. The creation of user-defined functions only separates registration and execution, resulting in abnormal failure during verification in the formal environment during execution, leading to production task failures and online problems.

[0092] In the data processing method provided in the first embodiment of this application, by receiving a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; according to the target processing method in the data processing request, a structured query statement for calling a first custom function is generated, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide the execution conditions of the target code; the first custom function is called from the target interface of the target database according to the structured query statement, and the target data is processed according to the first custom function to obtain a processing result, which solves the technical problem in the prior art that the user needs to perform operations such as compilation and packaging, adding resources, and creating functions before data can be processed through a custom function, resulting in relatively low data processing efficiency. In this solution, when data processing of target data is required, a structured query statement for calling the first custom function can be directly generated according to the current target processing method, and finally the first custom function is called from the target interface of the target database using the structured query statement to complete the processing of the target data, avoiding the need for the user to perform operations such as compilation and packaging, adding resources, and creating functions before data can be processed through a custom function. Moreover, the custom function in this solution is obtained based on a general programming language, which improves the writing efficiency compared to writing and encapsulating custom functions using SQL language in the prior art. In addition, the custom function in this solution supports creation once and usage multiple times, avoiding the need for the user to perform operations such as establishing a project locally, compiling, packaging, and releasing online, thereby achieving the effect of improving the data processing efficiency.

[0093] It should be noted that for the foregoing method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0094] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0095] Embodiment 2

[0096] According to an embodiment of the present application, there is also provided a data processing method, as Figure 6 shown, the method includes:

[0097] Step S601: Obtain a data processing request uploaded by a client. Among them, the data processing request at least includes: target data to be processed and a target processing method for the target data;

[0098] Step S602: In the cloud server, generate a structured query statement for calling a first custom function according to the target processing method in the data processing request. The first custom function is obtained based on a custom function template and a general programming language. The custom function template is used to provide the execution conditions of the target code. Call the first custom function from the target interface of the target database according to the structured query statement, and process the target data according to the first custom function to obtain a processing result.

[0099] Step S603: Return the processing result to the client.

[0100] In the cloud server, the specific method for processing the target data is the same as the method in Embodiment 1 and will not be elaborated here.

[0101] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0102] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present application.

[0103] Embodiment 3

[0104] According to an embodiment of the present application, there is also provided a data processing device for implementing the above data processing method, as Figure 7 shown. The device includes: a receiving unit 701, a first generating unit 702, and a calling unit 703.

[0105] The receiving unit 701 is configured to receive a data processing request, where the data processing request at least includes: target data to be processed and a target processing manner for the target data;

[0106] The first generating unit 702 is configured to generate a structured query statement for calling a first custom function according to the target processing manner in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions for the target code;

[0107] The calling unit 703 is configured to call the first custom function from the target interface of the target database according to the structured query statement, and process the target data according to the first custom function to obtain a processing result.

[0108] In the data processing device provided in Embodiment 3 of the present application, a data processing request is received by a receiving unit 701. Among them, the data processing request at least includes: target data to be processed and a target processing method for the target data; a first generating unit 702 generates a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions for target code; a calling unit 703 calls the first custom function from a target interface of a target database according to the structured query statement, and processes the target data according to the first custom function to obtain a processing result, solving the technical problem in the prior art that the user needs to perform operations such as compilation and packaging, resource addition, and function creation before processing data through a custom function, resulting in relatively low data processing efficiency. In this solution, when target data needs to be processed, a structured query statement for calling the first custom function can be directly generated according to the current target processing method, and finally the first custom function is called from the target interface of the target database using the structured query statement to complete the processing of the target data, avoiding the need for the user to perform operations such as compilation and packaging, resource addition, and function creation before processing data through a custom function. Moreover, the custom function in this solution is obtained based on a general programming language, improving the writing efficiency compared with writing and encapsulating custom functions through SQL language in the prior art. Additionally, the custom function in this solution supports being created once and used multiple times, avoiding the need for the user to perform operations such as establishing a project locally, compiling, packaging, and releasing online, thereby achieving the effect of improving data processing efficiency.

[0109] In the data processing device provided in Embodiment 3 of the present application, the device further includes: a second generating unit, configured to generate target code according to the general programming language and the target processing method before generating a structured query statement for calling a first custom function according to the target processing method in the data processing request; a third generating unit, configured to generate a first custom function according to the custom function template and the target code.

[0110] In the data processing device provided in Embodiment 3 of the present application, the third generating unit includes: an adding subunit, configured to add the target code to a target position in the custom function template to obtain an initial custom function; a setting subunit, configured to perform setting processing on target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage class keywords and update class keywords; an analyzing subunit, configured to perform analyzing processing on the processed initial custom function to obtain a first custom function.

[0111] In the data processing device provided in the third embodiment of the present application, the analysis subunit includes: a generation module configured to generate an abstract syntax tree according to the processed initial custom function; a first analysis module configured to perform error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result; a second analysis module configured to perform compilation analysis on the processed initial custom function to obtain a second analysis result; and a correction module configured to correct the processed initial custom function according to the first analysis result and the second analysis result to obtain the first custom function.

[0112] In the data processing device provided in the third embodiment of the present application, the generation module includes: a splitting sub-module configured to split the processed initial custom function to obtain a plurality of strings; a filtering sub-module configured to filter the plurality of strings to obtain processed strings; and a generation sub-module configured to generate an abstract syntax tree according to the processed strings.

[0113] In the data processing device provided in the third embodiment of the present application, the first analysis module includes: a first analysis sub-module configured to analyze the data processing process of the code in the first custom function according to the abstract syntax tree in a real execution environment-free manner to obtain a first analysis sub-result; a second analysis sub-module configured to physically analyze the data processing process of the code in the first custom function according to the abstract syntax tree to obtain a second analysis sub-result; and a determination sub-module configured to obtain a first analysis result according to the first analysis sub-result and the second analysis sub-result.

[0114] In the data processing device provided in the third embodiment of the present application, the second analysis module includes: an acquisition sub-module configured to obtain target code according to the processed initial custom function; a compilation sub-module configured to perform compilation on the target code in a real execution environment to obtain an execution result, where the real execution environment is the real execution environment of the first custom function; and a third analysis sub-module configured to perform error analysis according to the execution result to obtain a second analysis result.

[0115] In the data processing device provided in the third embodiment of the present application, the device further includes: a storage unit configured to persistently store the first custom function in a target database after obtaining the first custom function according to a custom function template and the target code; and an extension unit configured to perform extension processing on a target interface of the target database to call the first custom function through the extended target interface.

[0116] It should be noted that the above receiving unit 701, first generating unit 702, and calling unit 703 correspond to steps S201 to S203 in Embodiment 1. The functions of the three units are the same as those of the corresponding steps in terms of the implemented examples and application scenarios, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0117] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0118] Embodiment 4

[0119] An embodiment of the present application can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the above computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0120] Optionally, in this embodiment, the above computer terminal can be located in at least one of multiple network devices in a computer network.

[0121] In this embodiment, the above computer terminal can execute the program code of the following steps in the data processing method: receiving a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide the execution conditions of the target code; calling the first custom function from the target interface of the target database according to the structured query statement, and processing the target data according to the first custom function to obtain a processing result.

[0122] The above computer terminal can execute the program code of the following steps in the data processing method: before generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, the method further includes: generating target code according to the general programming language and the target processing method; generating the first custom function according to the custom function template and the target code.

[0123] The above computer terminal can execute the program code for the following steps in the data processing method: Generating a first custom function based on a custom function template and target code includes: adding the target code to a target position in the custom function template to obtain an initial custom function; performing setting processing on target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage class keywords and update class keywords; performing analysis processing on the processed initial custom function to obtain the first custom function.

[0124] The above computer terminal can execute the program code for the following steps in the data processing method: Performing analysis processing on the processed initial custom function to obtain the first custom function includes: generating an abstract syntax tree based on the processed initial custom function; performing error analysis on the code in the first custom function based on the abstract syntax tree to obtain a first analysis result; performing compilation analysis on the processed initial custom function to obtain a second analysis result; correcting the processed initial custom function based on the first analysis result and the second analysis result to obtain the first custom function.

[0125] The above computer terminal can execute the program code for the following steps in the data processing method: Generating an abstract syntax tree based on the processed initial custom function includes: splitting the processed initial custom function to obtain multiple strings; filtering the multiple strings to obtain processed strings; generating an abstract syntax tree based on the processed strings.

[0126] The above computer terminal can execute the program code for the following steps in the data processing method: Performing error analysis on the code in the first custom function based on the abstract syntax tree to obtain the first analysis result includes: analyzing the data processing process of the code in the first custom function based on the abstract syntax tree in a real execution environment-free state to obtain a first sub-analysis result; physically analyzing the data processing process of the code in the first custom function based on the abstract syntax tree to obtain a second sub-analysis result; obtaining the first analysis result based on the first sub-analysis result and the second sub-analysis result.

[0127] The above computer terminal can execute the program code for the following steps in the data processing method: Performing compilation analysis on the processed initial custom function to obtain the second analysis result includes: obtaining target code based on the processed initial custom function; performing compilation on the target code in a real execution environment, where the real execution environment is the real execution environment of the first custom function; performing error analysis based on the execution result to obtain the second analysis result.

[0128] The above computer terminal may execute the program code of the following steps in the data processing method: After obtaining the first custom function based on the custom function template and the target code, the method further includes: persistently storing the first custom function in the target database; performing an extension process on the target interface of the target database to call the first custom function through the extended target interface.

[0129] Optionally, Figure 8 is a structural block diagram of a computer terminal according to an embodiment of the present application. As Figure 8 shown, the computer terminal 10 may include: one or more ( Figure 8 only one is shown in the figure) processors 102, a memory 104. The computing terminal 10 may further include a storage controller for controlling and managing the memory 104 through the storage controller; the computing terminal 10 may further include a peripheral interface for connecting a radio frequency module, an audio module, a display screen, etc. through the peripheral interface.

[0130] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizes the above data processing method. The memory may include a high-speed random access memory, and may further include a non-volatile memory, such as one or more magnetic storage devices, a flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include a memory remotely set relative to the processor, and these remote memories may be connected to the terminal 10 through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The processor may call the information and application programs stored in the memory through the transmission device to execute the following steps: receiving a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; generating a structured query statement for calling the first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide the execution conditions of the target code; calling the first custom function from the target interface of the target database according to the structured query statement, and processing the target data according to the first custom function to obtain a processing result.

[0132] Optionally, the above-mentioned processor may also execute the program code of the following steps: Before generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, the method further includes: generating target code according to a general programming language and the target processing method; generating a first custom function according to a custom function template and the target code.

[0133] Optionally, the above-mentioned processor may also execute the program code of the following steps: Generating a first custom function according to a custom function template and the target code includes: adding the target code to a target position in the custom function template to obtain an initial custom function; performing setting processing on target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage class keywords and update class keywords; performing analysis processing on the processed initial custom function to obtain a first custom function.

[0134] Optionally, the above-mentioned processor may also execute the program code of the following steps: Performing analysis processing on the processed initial custom function to obtain a first custom function includes: generating an abstract syntax tree according to the processed initial custom function; performing error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result; performing compilation analysis on the processed initial custom function to obtain a second analysis result; correcting the processed initial custom function according to the first analysis result and the second analysis result to obtain a first custom function.

[0135] Optionally, the above-mentioned processor may also execute the program code of the following steps: Generating an abstract syntax tree according to the processed initial custom function includes: performing splitting processing on the processed initial custom function to obtain multiple strings; performing filtering processing on the multiple strings to obtain processed strings; generating an abstract syntax tree according to the processed strings.

[0136] Optionally, the above-mentioned processor may also execute the program code of the following steps: Performing error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result includes: analyzing the data processing process of the code in the first custom function according to the abstract syntax tree in a real execution environment-free manner to obtain a first analysis sub-result; performing physical analysis on the data processing process of the code in the first custom function according to the abstract syntax tree to obtain a second analysis sub-result; obtaining a first analysis result according to the first analysis sub-result and the second analysis sub-result.

[0137] Optionally, the above-mentioned processor may also execute the program code of the following steps: perform compilation analysis based on the processed initial custom function, and obtain a second analysis result, including: obtaining target code based on the processed initial custom function; in a real execution environment, perform compilation on the target code to obtain an execution result, where the real execution environment is the real execution environment of the first custom function; perform error analysis based on the execution result to obtain a second analysis result.

[0138] Optionally, the above-mentioned processor may also execute the program code of the following steps: after obtaining the first custom function based on the custom function template and the target code, the method further includes: persistently storing the first custom function in a target database; performing extension processing on a target interface of the target database to call the first custom function through the extended target interface.

[0139] Those of ordinary skill in the art can understand that Figure 8 the structure shown is only schematic, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and a mobile Internet device (Mobile Internet Devices, MID), a PAD and other terminal devices. Figure 8 It does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in Figure 8 or have a different configuration from that shown in Figure 8 shown.

[0140] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disc, etc.

[0141] Embodiment 5

[0142] An embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the above storage medium may be used to save the program code executed by the data processing method provided in the first embodiment above.

[0143] Optionally, in this embodiment, the above storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0144] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions for the target code; calling the first custom function from a target interface of the target database according to the structured query statement, and processing the target data according to the first custom function to obtain a processing result.

[0145] The above storage medium is configured to store program code for performing the following steps: before generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, the method further includes: generating target code according to the general programming language and the target processing method; generating a first custom function according to the custom function template and the target code.

[0146] The above storage medium is configured to store program code for performing the following steps: generating a first custom function according to the custom function template and the target code includes: adding the target code to a target position in the custom function template to obtain an initial custom function; performing setting processing on target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage class keywords and update class keywords; performing analysis processing on the processed initial custom function to obtain a first custom function.

[0147] The above storage medium is configured to store program code for performing the following steps: performing analysis processing on the processed initial custom function to obtain a first custom function includes: generating an abstract syntax tree according to the processed initial custom function; performing error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result; performing compilation analysis according to the processed initial custom function to obtain a second analysis result; correcting the processed initial custom function according to the first analysis result and the second analysis result to obtain a first custom function.

[0148] The above storage medium is configured to store program code for performing the following steps: generating an abstract syntax tree according to the processed initial custom function includes: performing splitting processing on the processed initial custom function to obtain multiple strings; performing filtering processing on the multiple strings to obtain processed strings; generating an abstract syntax tree according to the processed strings.

[0149] The above storage medium is configured to store program code for performing the following steps: performing error analysis on the code in the first custom function according to the abstract syntax tree, and obtaining a first analysis result, including: analyzing the data processing process of the code in the first custom function according to the abstract syntax tree in a real execution environment-free manner to obtain a first sub-analysis result; performing physical analysis on the data processing process of the code in the first custom function according to the abstract syntax tree to obtain a second sub-analysis result; and obtaining the first analysis result according to the first sub-analysis result and the second sub-analysis result.

[0150] The above storage medium is configured to store program code for performing the following steps: performing compilation analysis according to the processed initial custom function, and obtaining a second analysis result, including: obtaining target code according to the processed initial custom function; performing compilation on the target code in a real execution environment, where the real execution environment is the real execution environment of the first custom function, to obtain an execution result; and performing error analysis according to the execution result to obtain the second analysis result.

[0151] The above storage medium is configured to store program code for performing the following steps: after obtaining the first custom function according to the custom function template and the target code, the method further includes: persistently storing the first custom function in the target database; and performing extension processing on the target interface of the target database to call the first custom function through the extended target interface.

[0152] The serial numbers of the embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0153] In the above embodiments of the present application, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0154] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, and the indirect coupling or communication connection of the units or modules may be in an electrical or other form.

[0155] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0158] The above is only the preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A data processing method, characterized in that, Including: Receiving a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; Generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general - purpose programming language, and the custom function template is used to provide execution conditions for target code; Invoking the first custom function from a target interface of a target database according to the structured query statement, and processing the target data according to the first custom function to obtain a processing result.

2. The method according to claim 1, wherein Before generating a structured query statement for calling a first custom function according to the target processing method in the data processing request, the method further includes: Generating target code according to the general - purpose programming language and the target processing method; Generating the first custom function according to the custom function template and the target code.

3. The method according to claim 2, wherein Generating the first custom function according to the custom function template and the target code includes: Adding the target code to a target position in the custom function template to obtain an initial custom function; Performing setting processing on target keywords in the initial custom function to obtain a processed initial custom function, where the target keywords at least include storage - class keywords and update - class keywords; Performing analysis processing on the processed initial custom function to obtain the first custom function.

4. The method according to claim 3, wherein Performing analysis processing on the processed initial custom function to obtain the first custom function includes: Generating an abstract syntax tree according to the processed initial custom function; Performing error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result; Performing compilation analysis on the processed initial custom function to obtain a second analysis result; Correcting the processed initial custom function according to the first analysis result and the second analysis result to obtain the first custom function.

5. The method according to claim 4, wherein Generating an abstract syntax tree according to the processed initial custom function includes: Performing splitting processing on the processed initial custom function to obtain multiple strings; Performing filtering processing on the multiple strings to obtain processed strings; Generating the abstract syntax tree according to the processed strings.

6. The method according to claim 4, wherein Performing error analysis on the code in the first custom function according to the abstract syntax tree to obtain a first analysis result includes: Analyzing the data - processing process of the code in the first custom function according to the abstract syntax tree in a real - execution - environment - free manner to obtain a first sub - analysis result; Performing physical analysis on the data - processing process of the code in the first custom function according to the abstract syntax tree to obtain a second sub - analysis result; Obtaining the first analysis result according to the first sub - analysis result and the second sub - analysis result.

7. The method according to claim 4, characterized in that, Performing compilation analysis on the processed initial custom function to obtain a second analysis result includes: Obtain the target code according to the processed initial custom function; In a real execution environment, compile the target code to obtain an execution result, where the real execution environment is the real execution environment of the first custom function; Perform error analysis according to the execution result to obtain the second analysis result.

8. The method according to claim 4, characterized in that, After obtaining the first custom function according to the custom function template and the target code, the method further includes: Persistently store the first custom function in the target database; Perform extension processing on the target interface of the target database to call the first custom function through the extended target interface.

9. A data processing method, characterized in that Includes: Obtain a data processing request uploaded by a client, where the data processing request at least includes: target data to be processed and a target processing method for the target data; In a cloud server, generate a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions for the target code. Call the first custom function from the target interface of the target database according to the structured query statement, and process the target data according to the first custom function to obtain a processing result; Return the processing result to the client.

10. A data processing device, characterized in that, Includes: A receiving unit, configured to receive a data processing request, where the data processing request at least includes: target data to be processed and a target processing method for the target data; A first generating unit, configured to generate a structured query statement for calling a first custom function according to the target processing method in the data processing request, where the first custom function is obtained based on a custom function template and a general programming language, and the custom function template is used to provide execution conditions; A calling unit, configured to call the first custom function from the target interface of the target database according to the structured query statement, and process the target data according to the first custom function to obtain a processing result.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, where when the program runs, it controls the device where the storage medium is located to execute the data processing method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, Includes: A memory, storing an executable program; A processor, configured to run the program, where when the program runs, it executes the data processing method according to any one of claims 1 to 9.

Images