A method and system for implementing user-defined functions based on dynamic encapsulation

Abstract

The application provides a user-defined function implementation method and system based on dynamic packaging. The user business class is dynamically loaded, and the calling process of the specified business implementation method in the class is packaged in the Scala Function class to build a general Spark UDF for the user to submit Spark SQL, so that the user can develop the regular business according to the development specification and syntax structure of the user, without being limited by the Spark UDF writing rules and syntax constraints, reducing the learning and development cost of the user, and simultaneously releasing the binding of the business code and the Spark framework, so that the business code can be easily integrated into other components, improving the reusability of the business code.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software, and in particular to a user-defined function implementation method and system based on dynamic encapsulation. BACKGROUND

[0002] Almost all databases and distributed systems supporting SQL statements provide an extension interface for users to enhance the processing capability of SQL statements to adapt to complex and personalized business requirements. These extensions are called user-defined functions (UDF) to accept parameters, perform operations and return the results of the operation. Broadly speaking, UDF is a collection of user-defined scalar functions (UDF), user-defined table-valued functions (UDTF), and user-defined aggregate functions (UDAF). Narrowly speaking, UDF refers to user-defined scalar functions (UDF), which are used most frequently.

[0003] Apache Spark is the most widely used distributed computing component in various big data application scenarios. Spark internally preloads many general UDF functions. When these functions cannot support complex business implementation, users can extend Spark SQL functions by writing, registering and using custom functions to meet the needs of diversified data processing and business analysis.

[0004] However, currently, to extend SparkUDF, users usually need to first write the corresponding program according to the business requirements, comply with the writing rules and syntax constraints of SparkUDF development, and then compile and package it before uploading it to the Spark cluster together with the corresponding Spark job for submission and running. Through this way of developing UDF, users need to not only be familiar with their own business development specifications and syntax structures, but also need to understand the writing rules and syntax constraints of SparkUDF, which increases the learning and development cost of users. Since the implementation of UDF needs to rely on the data structure of Spark, users cannot simply apply the UDF business code implementation to other components, which also limits the reusability of the business code. SUMMARY

[0005] To solve the above technical problems, the present application provides a user-defined function implementation method based on dynamic encapsulation, comprising:

[0006] S1. Receive the business class file package and Spark SQL job request submitted by the user, and parse the Spark SQL job request. The Spark SQL job request includes the fully qualified class name of the business class, the name of the concrete implementation method of the business class, the Spark UDF name corresponding to the business class method, and the Spark SQL statement. The Spark SQL statement contains the Spark UDF name corresponding to the business class method.

[0007] S2. Based on the fully qualified class name of the business class, dynamically load the corresponding business class of the business class file package using reflection. Obtain the calling process, return value type, and input parameter definition of the specific implementation method of the business class based on the name of the specific implementation method of the business class. Encapsulate the execution function based on the calling process, return value type, and input parameter definition of the specific implementation method of the business class. Construct a Function class based on the execution function and the input parameter definition of the specific implementation method of the business class.

[0008] S3. Encapsulate the return type of the specific implementation method of the business class, the Function class, and the UDF input parameter sequence to be processed in a ScalaUDF class, and construct a Scala UDF object;

[0009] S4. Take the Scala UDF object as the return value and the UDF input parameter sequence to be processed as the parameter, and encapsulate it into a Spark UDF function.

[0010] S5. Register the Spark UDF name and Spark UDF function corresponding to the business class method to the SparkSQL runtime environment;

[0011] S6. Run the Spark SQL statement based on the preset execution module to call the registered Spark UDF function.

[0012] In some specific embodiments, the business class file package mentioned in step S1 includes a Jar package of the business class. Users can encapsulate business data processing logic in a business class according to their own development specifications and syntax structure. This class has no dependency on the Spark framework or data structure and does not need to inherit from the Spark UDF family classes, such as UDF1, UDF2, ..., UDF22, etc. The business class is then packaged into a Jar and submitted.

[0013] In some specific embodiments, step S2, which involves encapsulating the execution function based on the calling process, return value type, and input parameter definition of the specific implementation method of the business class, includes the following steps:

[0014] S21. Obtain the business class and the calling process of its specific implementation methods through lazy loading;

[0015] S22, by setting the input parameter type as the Scala sequence type, setting the return value type as AnyRef, and encapsulating the calling process of the business class implementation method, an execution function is constructed. The business class implementation method is delayed by lazy loading until actual use, avoiding the preset task execution unit from recognizing the method at runtime; by encapsulating the calling process of the business class implementation method, the business data processing process is completed at runtime.

[0016] In some specific embodiments, the step S2 of constructing the Function class based on the execution function and the input parameter definition of the business class implementation method comprises the following steps:

[0017] S23, the business processing process and the input parameter definition of the implementation method in the business class are obtained by reflection, and the input parameter definition includes the input parameter purpose and the input parameter order of the implementation method in the business class;

[0018] S24, based on the input parameter purpose, the type of the Function class to be encapsulated is determined, based on the input parameter purpose and the input parameter order of the implementation method in the business class, the input parameters of the Object type are set in the input parameter order, and the return value type of the Any type is set to construct a default method, and the Function class is constructed by encapsulating the default method and the exception handling structure block;

[0019] S25, the calling of the execution function is encapsulated in the default method of the Function class;

[0020] S26, the business processing process of the implementation method in the business class is encapsulated in the exception handling structure block of the default method. The calling of the execution function is encapsulated in the default method to complete the business data processing logic at runtime, the return value type of the Any type is set to construct the default method for returning the execution function calling result, and the business processing process of the implementation method in the business class is encapsulated in the exception handling structure block of the default method for corresponding exception capture and processing.

[0021] In some specific embodiments, in the step S21 of setting the input parameter type as the Scala sequence type, the number of elements in the Scala sequence type is limited to between 0 and 22. The number of elements in the Scala sequence type is constrained by the ScalaFunction family class.

[0022] In some specific embodiments, in the step S24 of determining the type of the Function class to be encapsulated based on the input parameter object, the input parameter object is limited between 0 and 22. The input parameter object is constrained by the ScalaFunction family class.

[0023] According to a second aspect of the present application, a system for implementing user-defined functions based on dynamic encapsulation is provided, which comprises:

[0024] a driving module configured to receive a business class file package submitted by a user and a Spark SQL job request, and parse the Spark SQL job request, the Spark SQL job request comprising a fully qualified class name of a business class, a specific implementation method name of the business class, a Spark UDF name corresponding to a business class method, and a Spark SQL statement, wherein the Spark SQL statement contains the Spark UDF name corresponding to the business class method; dynamically loading a corresponding business class of the business class file package by reflection based on the fully qualified class name of the business class, obtaining a calling process, a return value type and an input parameter definition of a business class specific implementation method based on the specific implementation method name of the business class, encapsulating an execution function based on the calling process, the return value type and the input parameter definition of the business class specific implementation method, and constructing a Function class based on the execution function and the input parameter definition of the business class specific implementation method; encapsulating the return value type of the business class specific implementation method, the Function class and a UDF input parameter sequence to be processed in a Scala UDF class, and constructing a Scala UDF object; encapsulating a Spark UDF function with the Scala UDF object as a return value and the UDF input parameter sequence to be processed as a parameter; and registering the Spark UDF name corresponding to the business class method and the Spark UDF function to a Spark SQL runtime environment.

[0025] an execution module configured to run the Spark SQL statement to call the registered Spark UDF function.

[0026] In some specific embodiments, the driving module comprises a Spark Driver, and the execution module comprises a Spark Executor, and the execution module is further configured to iteratively process the Spark UDF and the SQL operator until the end of a task. The Spark Driver is responsible for job scheduling, and the Spark Executor is responsible for executing specific computing tasks.

[0027] According to a third aspect of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation manner of the first aspect.

[0028] According to a fourth aspect of the present application, a computer readable storage medium is provided, having stored thereon a computer program, which when executed by a processor implements the method as described in any implementation manner of the first aspect.

[0029] The present application provides a user-defined function implementation method and system based on dynamic encapsulation, and the technical effect is that:

[0030] The present application provides a user-defined function implementation method and system based on dynamic encapsulation, and the technical effect is that: BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings are included to provide a further understanding of embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the present application. Other embodiments and many of the intended advantages of the present application will be readily appreciated as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings. The elements of the drawings are not necessarily to scale relative to each other. Like reference numerals designate corresponding similar parts.

[0032] Figure 1 is a flow chart of the user-defined function implementation method based on dynamic encapsulation of one embodiment of the present application;

[0033] Figure 2 is a flow chart of the Function class construction of one specific embodiment of the present application;

[0034] Figure 3 is a framework diagram of the user-defined function implementation system based on dynamic encapsulation of one specific embodiment of the present application;

[0035] Figure 4 is a framework diagram of the Function class encapsulation of one specific embodiment of the present application;

[0036] Figure 5 is an example diagram of a business class of a specific embodiment of the present application;

[0037] Figure 6 is an example diagram of a Spark SQL job request of a specific embodiment of the present application;

[0038] Figure 7 is a structural schematic diagram of a computer device of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely intended for the purpose of interpretation of the related application and are not limiting of the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for the purpose of description.

[0040] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0041] Figure 1 is a flowchart of a user-defined function implementation method based on dynamic packaging of an embodiment of the present application. As shown in Figure 1 , the method comprises:

[0042] S1, receiving a business class file package submitted by a user and a Spark SQL job request, and parsing the Spark SQL job request, the Spark SQL job request comprising a fully qualified class name of a business class, a specific implementation method name of the business class, a Spark UDF name corresponding to a business class method, and a Spark SQL statement, wherein the Spark SQL statement contains the Spark UDF name corresponding to the business class method.

[0043] In some specific embodiments, the business class file package of step S1 comprises a Jar package of the business class. The user can encapsulate the business data processing logic in the business class according to his own development specification and syntax structure, the class has no dependency relationship with the Spark framework or data structure, and does not need to inherit the SparkUDF family class such as UDF1, UDF2, …, UDF22, etc., and the business class is packaged as a Jar for submission.

[0044] S2, dynamically loading the corresponding business class of the business class file package through reflection based on the full-qualified class name of the business class, obtaining the calling process, return value type and input parameter definition of the business class specific implementation method based on the specific implementation method name of the business class, encapsulating an execution function based on the calling process, return value type and input parameter definition of the business class specific implementation method, and constructing a Function class based on the execution function and the input parameter definition of the business class specific implementation method.

[0045] In some specific embodiments, the Function class includes a Scala Function class, i.e., a function object of Scala.

[0046] In some specific embodiments, Figure 2 The flow chart of the Function class constructed in one specific embodiment of the present application is shown in FIG. 2. Figure 2 As shown in step S2, the execution function is encapsulated based on the calling process, return value type and input parameter definition of the business class specific implementation method, including the following steps:

[0047] S21, obtaining the business class and the calling process of the business class specific implementation method through lazy loading;

[0048] S22, setting the input parameter type as a Scala sequence type, setting the return value type as AnyRef, and encapsulating the calling process of the business class specific implementation method to construct an execution function. The business class specific implementation method is delayed to actual use through lazy loading, avoiding the situation that the method cannot be recognized by the preset task execution unit at runtime; the calling process of the business class specific implementation method is encapsulated to complete the business data processing process at runtime.

[0049] In the step S22 of setting the input parameter type as a Scala sequence type, the number of elements in the Scala sequence type is limited to between 0 and 22. The number of elements in the Scala sequence type is constrained by the Scala Function family class.

[0050] In some specific embodiments, the step S2 of constructing the Function class based on the execution function and the input parameter definition of the business class specific implementation method includes the following steps:

[0051] S23, obtaining the business processing process and input parameter definition of the specific implementation method in the business class through reflection, wherein the input parameter definition includes the number and order of input parameters of the specific implementation method in the business class;

[0052] S24, determining the type of the Function class to be encapsulated based on the number of the input parameters, setting the input parameters of the Object type in the order of the input parameter sequence and setting the return value of the Any type to construct the default method based on the number of the input parameters and the order of the input parameters of the specific implementation method in the business class, and encapsulating the default method and the exception handling structure block to construct the Function class.

[0053] In the step S24, the number of the input parameters is limited between 0 and 22, and the number of the input parameters is restricted by the ScalaFunction family class.

[0054] S25, encapsulating the call of the execution function in the default method of the Function class;

[0055] S26, encapsulating the business processing procedure of the specific implementation method in the business class in the exception handling structure block of the default method. The call of the execution function is encapsulated in the default method to complete the business data processing logic at runtime, the return value of the Any type is set to construct the default method for returning the execution function call result, and the business processing procedure of the specific implementation method in the business class is encapsulated in the exception handling structure block of the default method for corresponding exception capture and processing.

[0056] S3, encapsulating the return value type of the specific implementation method of the business class, the Function class and the UDF input parameter sequence to be processed in the ScalaUDF class, and constructing a Scala UDF object;

[0057] S4, encapsulating the Scala UDF object as the return value and the UDF input parameter sequence to be processed as the parameter into a Spark UDF function;

[0058] S5, registering the Spark UDF function to the SparkSQL runtime environment with the Spark UDF name corresponding to the business class method;

[0059] S6, running the Spark SQL statement based on the preset execution module to call the registered Spark UDF function.

[0060] With reference to Figure 3 , Figure 3A framework diagram of a dynamic packaging-based user-defined function implementation system according to an embodiment of the present application, the system architecture comprising: a driving module configured to receive a user-submitted business class file package and a Spark SQL job request, and parse the Spark SQL job request, the Spark SQL job request comprising a fully qualified class name of a business class, a specific implementation method name of the business class, a Spark UDF name corresponding to a business class method, and a Spark SQL statement, wherein the Spark SQL statement contains the Spark UDF name corresponding to the business class method; dynamically loading a corresponding business class of the business class file package through reflection according to the fully qualified class name of the business class, obtaining a call process, a return value type, and an input parameter definition of a business class specific implementation method based on the specific implementation method name of the business class, encapsulating an execution function based on the call process, the return value type, and the input parameter definition of the business class specific implementation method, and constructing a Function class based on the execution function and the input parameter definition of the business class specific implementation method; encapsulating the return value type of the business class specific implementation method, the Function class, and a sequence of UDF input parameters to be processed in a Scala UDF class, and constructing a Scala UDF object; encapsulating the Scala UDF object as a return value and the sequence of UDF input parameters to be processed as a parameter into a Spark UDF function; registering the Spark UDF name corresponding to the business class method and the Spark UDF function to a Spark SQL runtime environment; and an execution module configured to run the Spark SQL statement to call the registered Spark UDF function.

[0061] In some embodiments, the driving module comprises a Spark Driver, and the execution module comprises a Spark Executor, and the execution module is further configured to iteratively process a Spark UDF and a SQL operator until a task ends. The Spark Driver is responsible for job scheduling, and the Spark Executor is responsible for executing specific computing tasks.

[0062] In some embodiments, referring to Figure 3 , the user needs to previously perform writing and submission of the business class file package and the Spark SQL job request, which are specifically described as follows:

[0063] (1.1) Business class construction: the user encapsulates business data processing logic in a business class according to his own development specifications and syntax structures, and the class has no dependency relationship with a Spark framework or a data structure, and does not need to inherit Spark UDF family classes such as UDF1, UDF2, …, and UDF22.

[0064] In the above manner, the user does not need to understand the SparkUDF writing rules and syntax constraints, and only needs to focus on the implementation of the specific business processing logic, thereby reducing the learning and development cost of the user, and also avoiding the binding of the business code and the Spark framework, so that the business code can be easily integrated into other components, and the reusability of the business code is improved.

[0065] (1.2) Construction of Spark SQL job request: the user constructs a Spark SQL job request. The request parameters include the fully qualified class name of the business class, the specific implementation method name of the business class (such as: apply), the SparkUDF name corresponding to the business class method (that is, the UDF method name to be used in Spark SQL), and the Spark SQL statement to be requested, and the SQL statement is processed by the above Spark UDF name.

[0066] (1.3) Submit Spark SQL job

[0067] The user submits the business class file package generated by compiling and packaging, that is, the business class Jar package, and the above constructed Spark SQL request to the Web service of the Spark Driver for business processing.

[0068] Continuing to refer to Figure 3 , the Spark runtime architecture mainly consists of two important components: Spark Driver and Spark Executor. The Spark Driver is responsible for job scheduling, and the Spark Executor is responsible for executing specific computing tasks. The SparkDriver realizes the functions of parsing request parameters, loading business classes, obtaining the return value type of the implementation method in the business class, encapsulating Function, constructing Scala UDF, encapsulating Spark UDF, and registering Spark UDF. The specific description is as follows:

[0069] (2.1) Parsing request parameters

[0070] The SparkDriver is integrated in the Web service, and after receiving the user request, the Spark SQL job parameters are parsed, which include the business class and UDF related parameters in (1.2).

[0071] (2.2) Load business class

[0072] The business class is dynamically loaded according to the fully qualified class name of the business class through reflection.

[0073] (2.3) Obtain the return value type of the implementation method in the business class

[0074] Identify the implementation method in the business class, and then get the return value type of the method. If the user does not specify the method name, use apply as the default method.

[0075] (2.4) encapsulate Function, which will be used to build Scala UDF, so that it can be registered as SparkUDF and applied in Spark SQL statement. Refer to Figure 4 The encapsulated process is as follows:

[0076] (2.4.1) Get the implementation method in the business class: get the business class and its implementation method through lazy loading, and delay the loading of the implementation method to the actual use, so as to avoid the situation that Spark Executor runtime cannot recognize the method.

[0077] (2.4.2) Build execution function: the execution function will be called in the apply method body of Function class and return the processing result. By setting the input parameter type of the execution function as Seq[Object], the number of elements in Seq is restricted by ScalaFunction family class, limited to 0-22. In the method body of the execution function, the implementation method in the business class obtained from (2.4.1) is called to complete the business data processing process. Set the return value type of the execution function as AnyRef, so that the function can return the user-defined data type.

[0078] (2.4.3) encapsulate Function: get the number of input parameters of the specific implementation method in the business class through reflection, and the number of parameters is restricted by ScalaFunction family class, limited to 0-22; according to the number of input parameters, determine the type of Function to be encapsulated. The internal implementation of Spark UDF is written in Scala, and the function implementation in Scala language is mainly based on Function family, which has 23 Function Traits numbered from 0 to 22, where the number represents the number of Function input parameters, such as Function0, Function1,..., Function22; in the Function class, implement the default method apply, and define N Object input parameters for the apply method according to the number of input parameters obtained through reflection (such as: N).

[0079] In the apply method body, the execution function constructed in (2.4.2) is directly called to complete the business data processing logic. The return value type of the apply method is set to Any, which is used to return the function call result. The apply method encapsulates the business processing process in an exception handling block to perform corresponding exception capture and processing.

[0080] (2.5) Constructing Scala UDF

[0081] The return value type of the implementation method in the business class obtained in (2.3), the Function constructed in (2.4), and the UDF input parameter sequence to be processed are encapsulated in a Scala UDF class that meets the requirements of the Spark UDF framework, and a ScalaUDF object is constructed.

[0082] (2.6) Encapsulating SparkUDF

[0083] The ScalaUDF object is encapsulated into a SparkUDF function as a return value, and the UDF input parameter sequence to be processed is used as a parameter.

[0084] (2.7) Registering SparkUDF

[0085] The SparkUDF name corresponding to the business class method obtained in (1.2) and the SparkUDF encapsulated in (2.6) are registered to the Spark service for use by the Spark SQL runtime.

[0086] As an example, refer to Figure 5 , the user writes a business class, where the business processing process is encapsulated in the apply method, which receives a string and returns a string value after processing. Refer to Figure 6 , after the user compiles and packages the business class, a Spark SQL job request is constructed, including the fully qualified class name of the business class (com.example.MyUDF), the specific implementation method name of the business class (apply), the SparkUDF name corresponding to the business class method (echoHi), and the Spark SQL statement to be requested. In the SQL statement, the specified field content is processed using the Spark UDF. After the SparkSQL execution is completed, the field processing result is "Hi, Spark UDF".

[0087] Through the above manner, the user business class is dynamically loaded, and the calling process of the specified business implementation method in the class is encapsulated in the Scala Function class, a general Spark UDF is constructed for the user to submit the Spark SQL, so that the user can develop the regular business according to the development specification and syntax structure of the user, and is not limited by the Spark UDF writing rules and syntax constraints, reduces the learning and development cost of the user, meanwhile, the binding between the business code and the Spark framework is released, so that the business code can be easily integrated into other components, and the reusability of the business code is improved.

[0088] Reference will now be made to the following description Figure 7 which shows a structural schematic diagram of a computer system 700 suitable for implementing the electronic device of the embodiments of the present application. Figure 7 The electronic device shown is merely an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0089] As shown in Figure 7 , the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or programs loaded from a storage portion 709 into a random access memory (RAM) 704. In the RAM 704, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 704 are connected to each other through a bus 705. An input / output (I / O) interface 706 is also connected to the bus 705.

[0090] The following components are connected to the I / O interface 706: an input portion 707 including a keyboard, a mouse, and the like; an output portion 708 including a liquid crystal display (LCD), a speaker, and the like; a storage portion 709 including a hard disk, and the like; and a communication portion 710 including a network interface card such as a LAN card, a modem, and the like. The communication portion 710 performs communication processing via a network such as the Internet. A drive 711 is also connected to the I / O interface 706 as necessary. A removable recording medium 712 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 711 as necessary, so that a computer program read therefrom is installed in the storage portion 709 as necessary.

[0091] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable storage medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 710, and / or installed from the removable medium 712. When the computer program is executed by the central processing unit (CPU) 701, the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable storage medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable storage medium that can be used to carry or store computer readable program code except the computer readable storage medium that can be embodied as a computer readable program code means which can be directed to a transitory propagating signal per se. The program code contained on the computer readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0092] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0093] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0094] The modules involved in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware.

[0095] As another aspect, the present application also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: receive a business class file package submitted by a user and a Spark SQL job request, and parse the Spark SQL job request, the Spark SQL job request including a fully qualified class name of a business class, a specific implementation method name of the business class, a Spark UDF name corresponding to a business class method, and a Spark SQL statement, wherein the Spark UDF name corresponding to the business class method is included in the Spark SQL statement; dynamically load a corresponding business class of the business class file package by reflection according to the fully qualified class name of the business class, obtain a calling process, a return value type, and an input parameter definition of a business class specific implementation method based on the specific implementation method name of the business class, encapsulate an execution function based on the calling process, the return value type, and the input parameter definition of the business class specific implementation method, and construct a Function class based on the execution function and the input parameter definition of the business class specific implementation method; encapsulate the return value type of the business class specific implementation method, the Function class, and a UDF input parameter sequence to be processed in a ScalaUDF class, and construct a Scala UDF object; encapsulate the Scala UDF object as a return value and the UDF input parameter sequence to be processed as a parameter into a SparkUDF function; register the Spark UDF name corresponding to the business class method and the SparkUDF function to a Spark SQL runtime environment; and run the Spark SQL statement based on a preset execution module to call the registered SparkUDF function.

[0096] The above description is merely the preferred embodiments of the present application and the explanation of the applied technical principles. It should be understood by those skilled in the art that the inventive scope of the present application is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features can be replaced with the technical features disclosed in the present application (but not limited to) having similar functions to form the technical solutions.

Claims

1. A method for implementing user-defined functions based on dynamic encapsulation, characterized in that, include: S1. Receive the business class file package and Spark SQL job request submitted by the user, and parse the Spark SQL job request. The Spark SQL job request includes the fully qualified class name of the business class, the name of the specific implementation method of the business class, the Spark UDF name corresponding to the business class method, and the Spark SQL statement. The Spark SQL statement contains the Spark UDF name corresponding to the business class method. S2. Based on the fully qualified class name of the business class, dynamically load the corresponding business class from the business class file package using reflection. Obtain the call process, return type, and input parameter definitions of the specific implementation methods of the business class based on their names. Encapsulate the execution function based on the call process, return type, and input parameter definitions of the specific implementation methods of the business class. Obtain the business processing process and input parameter definitions of the specific implementation methods in the business class using reflection. The input parameter definitions include the input parameter items and their order in the specific implementation methods of the business class. Determine the type of the Function class to be encapsulated based on the input parameter items. Based on the input parameter items and their order in the specific implementation methods of the business class, set the input parameters of type Object according to the input parameter order, and set the return type of type Any to construct a default method. Construct the Function class by encapsulating the default method and an exception handling structure block. Encapsulate the call to the execution function in the default method of the Function class. The business processing procedures of the specific implementation methods in the business class are encapsulated in the exception handling structure block of the default method; S3. Encapsulate the return type of the specific implementation method of the business class, the Function class, and the UDF input parameter sequence to be processed in a ScalaUDF class, and construct a ScalaUDF object; S4. Take the Scala UDF object as the return value and the UDF input parameter sequence to be processed as the parameter, and encapsulate it into a Spark UDF function. S5. Register the Spark UDF name and Spark UDF function corresponding to the business class method to the Spark SQL runtime environment; S6. Run the Spark SQL statement based on the preset execution module to call the registered Spark UDF function.

2. The method for implementing user-defined functions based on dynamic encapsulation according to claim 1, characterized in that, The business class file package mentioned in step S1 includes the business class Jar package.

3. The method for implementing user-defined functions based on dynamic encapsulation according to claim 1, characterized in that, Step S2 describes the encapsulation and execution function based on the calling process, return value type, and input parameter definition of the specific implementation method of the business class. Includes the following steps: S21. Obtain the business class and the calling process of its specific implementation methods through lazy loading; S22. By setting the input parameter type to Scala sequence type, setting the return value type to AnyRef, and encapsulating the calling process of the specific implementation method of the business class, an execution function is constructed.

4. The method for implementing user-defined functions based on dynamic encapsulation according to claim 3, characterized in that, In step S21, the input parameter type is set to a Scala sequence type, and the number of elements in the Scala sequence type is limited to between 0 and 22.

5. The method for implementing user-defined functions based on dynamic encapsulation according to claim 1, characterized in that, In determining the type of the Function class to be encapsulated based on the input parameters, the number of input parameters is limited to between 0 and 22.

6. A system for implementing user-defined functions based on dynamic encapsulation, characterized in that, The system includes: The driver module is configured to receive user-submitted business class file packages and Spark SQL job requests, and parse the Spark SQL job requests. The Spark SQL job requests include the fully qualified class name of the business class, the name of the concrete implementation method of the business class, the name of the Spark UDF corresponding to the business class method, and the Spark SQL statement. The Spark SQL statement contains the Spark UDF name corresponding to the business class method. Based on the fully qualified class name of the business class, the module dynamically loads the corresponding business class from the business class file package using reflection. Based on the name of the concrete implementation method of the business class, it obtains the call process, return type, and input parameter definitions of the concrete implementation method. Based on the call process, return type, and input parameter definitions of the concrete implementation method, it encapsulates an execution function. It then obtains the business processing process and input parameter definitions of the concrete implementation method in the business class using reflection. The input parameter definitions include the input parameter names and their order in the concrete implementation method of the business class. Based on the input parameter names, it determines the Function class to be encapsulated. Based on the input parameters and their order in the specific implementation method of the business class, the input parameters are set to Object type according to the input parameter order, and the return value type is set to Any to construct a default method. The default method and an exception handling structure block are encapsulated to construct a Function class. The call to the execution function is encapsulated in the default method of the Function class. The business processing of the specific implementation method in the business class is encapsulated in the exception handling structure block of the default method. The return value type of the specific implementation method of the business class, the Function class, and the UDF input parameter sequence to be processed are encapsulated in a Scala UDF class, and a ScalaUDF object is constructed. The ScalaUDF object is used as the return value, and the UDF input parameter sequence to be processed is used as the parameter, encapsulated as a SparkUDF function. The SparkUDF name corresponding to the business class method and the Spark UDF function are registered to the Spark SQL runtime environment. The execution module is configured to run the Spark SQL statement to invoke the registered SparkUDF function.

7. A system for implementing user-defined functions based on dynamic encapsulation according to claim 6, characterized in that, The driving module includes Spark Driver, and the execution module includes Spark Executor. The execution module is also configured to iteratively process Spark UDFs and SQL operators until the task ends.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images