A method for parsing FlinkUDF functions from a JAR file
By using the ASM bytecode parsing engine to parse JAR files and directly manipulate bytecode, the problems of high resource consumption, long processing time, and dependencies in traditional methods are solved, achieving efficient UDF function parsing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-03
- Publication Date
- 2026-03-10
AI Technical Summary
Traditional methods for parsing JAR files suffer from high system resource consumption, excessive parsing time, and JAR file dependencies, leading to memory overflow and low parsing efficiency.
The ASM bytecode parsing engine is used to parse JAR files. By converting class files in bytecode form into binary byte streams and arrays, the underlying bytecode files can be directly manipulated to find and extract UDF function-related information, thus avoiding the use of class loaders.
It solves the problems of excessive memory consumption, long parsing time, and JAR file dependencies, improves parsing efficiency and speed, and reduces memory consumption and dependency loading.
Smart Images
Figure CN115686528B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of information processing technology, specifically relating to a method for parsing FlinkUDF functions from JAR files. Background Technology
[0002] User-defined functions (UDFs) are a crucial feature of the Flink framework, significantly extending the expressive power of Flink SQL. Commonly used UDFs fall into three categories: scalar functions, table functions, and aggregate functions.
[0003] User-defined functions (UDFs) in Flink are stored as JAR files. Therefore, if you need to obtain the UDF information contained in a JAR file, you need to parse the JAR file.
[0004] The traditional JAR file parsing process is as follows: 1) Write the relevant code in a local editor and compile it into a class file; 2) Package the class file into a JAR file; 3) Load the JAR file using a class loader and parse the relevant information of the UDF functions contained in the JAR file.
[0005] Traditional methods for loading and parsing JAR files using class loaders have the following drawbacks and shortcomings:
[0006] (1) High system resource consumption: When using a class loader to load classes, they need to be loaded into memory. When the JAR file is very large, the Java process will continuously perform GC, which can lead to memory overflow in severe cases.
[0007] (2) Long parsing time: When the class file is too large and the number of UDF functions is small, the parsing time of the JAR file will be too long, which will affect the user experience.
[0008] (3) JAR file dependency: During the process of loading a JAR file, the class loader needs to introduce additional dependencies, which may cause the JAR file to fail to load. Summary of the Invention
[0009] To address the shortcomings of existing technologies, this invention provides a method for parsing FlinkUDF functions from JAR files, which can effectively solve the aforementioned problems.
[0010] The technical solution adopted in this invention is as follows:
[0011] This invention provides a method for parsing FlinkUDF functions from a JAR file, comprising the following steps:
[0012] Step 1: Read the JAR file that needs to be parsed;
[0013] Step 2: Parse the read JAR file to obtain class files; the class files are bytecode format class files;
[0014] Step 3: Transfer the bytecode class file to the ASM bytecode parsing engine;
[0015] Step 4: The ASM bytecode parsing engine parses the class file in bytecode form. The specific method includes:
[0016] Step 4.1: The ASM bytecode parsing engine reads the bytecode class file into memory and converts it into a binary byte stream class file;
[0017] Step 4.2: The ASM bytecode parsing engine converts the binary byte stream class file into an array-like class file.
[0018] Step 4.3: The ASM bytecode parsing engine locates the parent class of the UDF and functions of all types in the class file in array form;
[0019] Step 4.4: The ASM bytecode parsing engine filters all types of functions obtained and retains UDF functions;
[0020] Step 4.5: The ASM bytecode parsing engine parses the UDF parent class obtained in step 4.3, obtains the content associated with the use of the UDF function, and extracts the associated content.
[0021] The ASM bytecode parsing engine parses the UDF function obtained in step 4.4, and obtains the input parameters and return value information contained in the UDF function. Then, it combines and packages the extracted related content with the input parameters and return value information to form a new UDF function.
[0022] Step 4.4: Return the new UDF function to the requester.
[0023] Preferably, step 4.2 specifically includes:
[0024] The binary byte stream file consists of n sequentially arranged binary bytes, represented as: a1, a2, ..., a n ;
[0025] Create an array B with n positions. Store the binary byte a1 into the first element position of array B to form the first element of array B, denoted as B[1]; store the binary byte a2 into the second element position of array B to form the second element of array B, denoted as B[2], and so on, storing the binary byte a1 into the first element position of array B to form the second element of array B, denoted as B[2]. n The nth element of array B is stored in the nth element position, forming the nth element of array B, denoted as B[n]. This converts the binary byte stream class file into an array class file.
[0026] Preferably, between steps 4.3 and 4.4, the following is also included:
[0027] After finding the UDF parent class and all types of functions from the class file in array form, extract the UDF parent class and all types of functions in binary byte stream form from the class file in array form;
[0028] The extracted binary byte stream of the UDF parent class and all types of functions is converted into strings to obtain the UDF parent class and all types of functions in string form; then, using the string form of the UDF parent class and all types of functions as the processing objects, step 4.4 is executed again.
[0029] Preferably, the string is a UTF-8 encoded string.
[0030] The method for parsing FlinkUDF functions from JAR files provided by this invention has the following advantages:
[0031] 1) This invention solves the problem of excessive memory usage when parsing JAR files.
[0032] 2) This invention solves the problem of excessive time consumption when parsing JAR files.
[0033] 3) This invention solves the problem of needing to load the dependencies when JAR files depend on each other. Attached Figure Description
[0034] Figure 1 This is a flowchart illustrating a method for parsing FlinkUDF functions from a JAR file, as provided by the present invention. Detailed Implementation
[0035] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.
[0036] This invention provides a method for parsing FlinkUDF functions from a JAR file, referencing... Figure 1 This includes the following steps:
[0037] Step 1: Read the JAR file that needs to be parsed;
[0038] Specifically, the user uploads the JAR file required to execute Flink SQL to the network storage service. This invention reads the JAR file from the network storage service and performs subsequent parsing steps. After the parsing is completed, the parsed UDF function is returned to the user.
[0039] Step 2: Parse the read JAR file to obtain class files; files ending with the .class extension are selected as class files. These class files are bytecode format.
[0040] Step 3: The bytecode class file is handed over to the ASM bytecode parsing engine; the ASM bytecode parsing engine receives the bytecode class file through an interface that uses Java ASM technology to optimize the reading of the class file.
[0041] Step 4: The ASM bytecode parsing engine parses the class file in bytecode form. The specific method includes:
[0042] Step 4.1: The ASM bytecode parsing engine reads the bytecode class file into memory and converts it into a binary byte stream class file;
[0043] Step 4.2: The ASM bytecode parsing engine converts the binary byte stream class file into an array-like class file.
[0044] This step is specifically as follows:
[0045] The binary byte stream file consists of n sequentially arranged binary bytes, represented as: a1, a2, ..., a n ;
[0046] Create an array B with n positions. Store the binary byte a1 into the first element position of array B to form the first element of array B, denoted as B[1]; store the binary byte a2 into the second element position of array B to form the second element of array B, denoted as B[2], and so on, storing the binary byte a1 into the first element position of array B to form the second element of array B, denoted as B[2]. n The nth element of array B is stored in the nth element position, forming the nth element of array B, denoted as B[n]. This converts the binary byte stream class file into an array class file.
[0047] By using arrays to represent and store class files in binary byte stream format, any position in the class file can be quickly accessed directly through array subscripts, and the underlying bytecode file can be directly manipulated in this way.
[0048] Step 4.3: The ASM bytecode parsing engine locates the parent class of the UDF and functions of all types in the class file in array form;
[0049] In the specific implementation, after finding the UDF parent class and all types of functions from the class file in array form, the UDF parent class and all types of functions in binary byte stream form are extracted from the class file in array form;
[0050] The extracted binary byte stream of the UDF parent class and all types of functions is converted into a string, for example, a UTF-8 encoded string, to obtain the UDF parent class and all types of functions in string form; then, using the string form of the UDF parent class and all types of functions as the processing object, step 4.4 is executed again.
[0051] As a specific implementation, after obtaining the class file in array format, since bytecode has a fixed format, referring to the bytecode file structure standard, the parent class of the UDF can be located by class name. The parent classes of UDFs include three types: AggregateFunction, ScalarFunction, and TableFunction. Functions of all types can be located by function name; each function has information such as function name, function parameters, and function return value. The parent classes of the UDFs and all types of functions are saved for further parsing by the ASM bytecode parsing engine.
[0052] In this invention, to avoid parsing costs, classes in the class library are not parsed. Since bytecode is manipulated directly, no additional class dependencies are needed, and users do not need to package and upload dependency files. Because bytecode files are parsed directly, unnecessary memory usage caused by class loading is avoided. Since a class loader is not used, the parsing speed is significantly faster.
[0053] Step 4.4: The ASM bytecode parsing engine filters all types of functions obtained and retains UDF functions;
[0054] Step 4.5: The ASM bytecode parsing engine parses the UDF parent class obtained in step 4.3, obtains the content associated with the use of the UDF function, and extracts the associated content.
[0055] The ASM bytecode parsing engine parses the UDF function obtained in step 4.4, and obtains the input parameters and return value information contained in the UDF function. Then, it combines and packages the extracted related content with the input parameters and return value information to form a new UDF function.
[0056] Step 4.4: Return the new UDF function to the requester.
[0057] Compared with existing technologies, the method for parsing FlinkUDF functions from JAR files provided by this invention has the following advantages:
[0058] 1) This invention solves the problem of excessive memory usage when parsing JAR files.
[0059] 2) This invention solves the problem of excessive time consumption when parsing JAR files.
[0060] 3) This invention solves the problem of needing to load the dependencies when JAR files depend on each other.
[0061] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for parsing FlinkUDF functions from JAR files, characterized in that, It comprises the following steps: Step 1, reading the JAR file to be parsed; Step 2, parsing the read JAR file to obtain a class file; the class file is a class file in bytecode form; Step 3, transferring the class file in bytecode form to an ASM bytecode parsing engine; Step 4, the ASM bytecode parsing engine performs parsing processing on the class file in bytecode form, and the specific method comprises: Step 4.1, the ASM bytecode parsing engine reads the class file in bytecode form into memory and converts it into a class file in binary byte stream form; Step 4.2, the ASM bytecode parsing engine converts the class file in binary byte stream form into an array form class file; Step 4.2 specifically comprises: The class file in the form of binary byte stream, including n binary bytes arranged in sequence, respectively represented as: a1, a2, …, a n ; An array B with n positions is created, the binary byte a1 is stored in the first element position of the array B, forming the first element of the array B, denoted as B[1]; the binary byte a2 is stored in the second element position of the array B, forming the second element of the array B, denoted as B[2], and so on, the binary byte a n is stored in the n-th element position of the array B, forming the n-th element of the array B, denoted as B[n], thus realizing the conversion of the class file in the form of the binary byte stream into the class file in the form of the array. Step 4.3, the ASM bytecode parsing engine finds the UDF parent class and all types of functions in the array form class file; Step 4.4, the ASM bytecode parsing engine filters all types of functions obtained to retain UDF functions; Step 4.5, the ASM bytecode parsing engine parses the UDF parent class obtained in step 4.3 to obtain associated content used with the UDF function, and extracts the associated content; The ASM bytecode parsing engine parses the UDF function obtained in step 4.4 to obtain input parameter and return value information contained in the UDF function, and then combines and packages the extracted associated content with the input parameter and return value information to form a new UDF function; Step 4.6, returning the new UDF function to the requester.
2. The method for parsing FlinkUDF function from JAR file according to claim 1, wherein, Between step 4.3 and step 4.4, it also comprises: After finding the UDF parent class and all types of functions from the array form class file, the binary byte stream form UDF parent class and all types of functions are intercepted from the array form class file; The intercepted binary byte stream form UDF parent class and all types of functions are converted into strings to obtain string form UDF parent class and all types of functions; then, the string form UDF parent class and all types of functions are taken as processing objects, and step 4.4 is executed again.
3. The method for parsing FlinkUDF function from JAR file according to claim 2, characterized in that, The string is a utf-8 encoded string.
Citation Information
Patent Citations
Data processing method and device
CN113312053A
User-defined function implementation method and device thereof, computer equipment and storage medium
CN113504904A