Method and system for dynamically constructing UDF based on large model

Abstract

The invention discloses a method for dynamically constructing a UDF based on a large model. The method comprises the following steps: S1, constructing a multi-dimensional requirement cue word of a user-defined function according to characteristics and use requirements of a target platform; s2, the multi-dimensional requirement cue words are input into a pre-trained large language model for code generation, the large language model is subjected to multi-round iteration based on the multi-dimensional requirement cue words, and an optimal code sequence for achieving the user-defined function is output; s3, code style standardization is carried out on the optimal code sequence, and a compiler or an interpreter and a static analysis tool are respectively called to carry out grammar check and static analysis on the optimal code sequence, and constructing an optimization cue word according to an error log output by the compiler or the interpreter and a static analysis result output by the static analysis tool, and inputting the optimization cue word into the large language model to carry out iterative optimization on the optimal code sequence.

Description

Technical Field

[0001] This invention relates to the field of large model technology, specifically to a method and system for dynamically constructing UDFs based on large models. Background Technology

[0002] In today's information age, data has become one of the core resources driving social development, and the importance of big data processing technology is increasingly prominent. With the exponential growth of data scale and the diversification of application scenarios, traditional data processing methods are no longer sufficient to meet real-world needs. Therefore, big data processing frameworks, represented by Flink, have emerged and developed rapidly. Within these frameworks, User-Defined Functions (UDFs), as an extension mechanism, provide users with flexible customization capabilities, enabling them to implement complex data processing logic according to specific business needs. This allows for the efficient execution of complex data transformation and analysis tasks, not only improving the flexibility and efficiency of data processing but also enabling deeper data mining to support more accurate decision-making. Furthermore, UDFs enable non-standardized data operations, compensating for the shortcomings of general processing frameworks in specific application scenarios, and are an indispensable tool for achieving personalized data analysis.

[0003] Meanwhile, large-scale intelligent agent technology has made significant progress in recent years. Through pre-training on massive amounts of data, large models possess powerful language understanding and generation capabilities, demonstrating strong potential, particularly in natural language processing and code generation. This rapid development of technology has opened up new possibilities for the automated generation of UDFs (User-Defined Functions).

[0004] While User-Defined Functions (UDFs) play a crucial role in big data processing, traditional UDF development methods still suffer from several limitations. First, UDF development heavily relies on professional programmers, creating a significant technical barrier for ordinary users. Non-professional users often need to invest considerable time learning programming languages ​​and related tools to develop and deploy UDFs. Second, traditional UDF development methods are inefficient, especially when facing complex data processing tasks. Manually writing code is not only time-consuming and labor-intensive but also prone to introducing human error. Furthermore, current technologies are insufficient in improving the efficiency and accessibility of big data processing, failing to meet the ever-increasing demands for data processing. These issues limit the widespread application of big data technologies, necessitating a more efficient and intelligent solution. Summary of the Invention

[0005] To address these issues, this invention proposes a method and system for dynamically constructing UDFs based on large models.

[0006] According to one aspect of the present invention, a method for dynamically constructing UDFs based on large models is proposed, comprising the following steps:

[0007] S1. Construct multi-dimensional requirement prompts for user-defined functions based on the characteristics and usage requirements of the target platform. The multi-dimensional requirement prompts include function description prompts, input / output requirement prompts, and auxiliary information prompts.

[0008] S2, the multi-dimensional requirement prompts are input into a pre-trained large language model to generate code. The large language model performs multiple iterations based on the multi-dimensional requirement prompts and outputs the optimal code sequence to implement the user-defined function.

[0009] S3, perform code style standardization on the optimal code sequence, and call the compiler or interpreter and static analysis tool to perform syntax checking and static analysis on the optimal code sequence respectively. Construct optimization prompt words based on the error logs output by the compiler or interpreter and the static analysis results output by the static analysis tool, and input the optimization prompt words into the large language model to iteratively optimize the optimal code sequence.

[0010] Specifically, S1 includes:

[0011] Based on the usage requirements, determine the function type of the user-defined function and construct corresponding function description prompts;

[0012] Based on the characteristics of the target platform, obtain the data types and language types supported by the target platform, determine the input parameter types and output result data types of the user-defined function based on the data types and language types supported by the target platform, and construct the input / output requirement prompt words;

[0013] Based on the characteristics and usage requirements of the target platform, determine whether the user-defined function needs to support parallel computing, whether a performance optimization threshold needs to be set, whether it has dependency library restrictions, whether it has a caching strategy, as well as the compatibility with the platform version and the requirements for exception handling, and construct the auxiliary information prompt words.

[0014] Designing effective prompts is crucial for guiding an agent to generate correct UDFs. The clarity, completeness, and relevance of the prompts directly impact the quality of the model's output. Clarity requires the prompts to be concise and unambiguous, ensuring the model accurately understands the user's intent. Completeness emphasizes that the prompts should include all necessary information, such as functional descriptions and input / output requirements, to reduce the likelihood of the model generating incorrect codes. Relevance means that the prompts need to be customized according to the specific task, ensuring the model focuses on the core requirements.

[0015] Specifically, the functional types of the user-defined functions include scalar functions, table-valued functions, aggregate functions, and window functions.

[0016] Specifically, the large language model described in S2, based on the multi-dimensional requirement prompt words, undergoes multiple rounds of iteration, including:

[0017] S201, the large language model generates an initial code sequence based on the multi-dimensional requirement prompt words;

[0018] S202, the large language model generates multiple candidate code sequences and corresponding candidate code sequence scores based on the initial code sequence, and retains a predetermined number of candidate code sequences as the current candidate code sequences based on the candidate code sequence scores;

[0019] S203, the large language model generates a new candidate code sequence based on the current candidate code sequence and updates the corresponding candidate code sequence score, and retains a predetermined number of candidate code sequences as new current candidate code sequences based on the candidate code sequence score;

[0020] S204, the large language model repeats S203 for multiple rounds of iteration until the termination condition is met;

[0021] S205, after the current candidate code sequence generates an end marker, it is treated as a completed sequence. The candidate code sequence scores of the completed sequences that meet the termination conditions are obtained, processed, and filtered, and the optimal code sequence is output.

[0022] During the process of generating code word by word (or token by token), the model maintains multiple candidate sequences in parallel. At each step, it retains several partial sequences with the highest probability, and finally selects the one with the highest comprehensive score from these complete sequences as the output. This avoids local optima and can explore code paths with more complete semantics and more reasonable structure.

[0023] Specifically, satisfying the termination condition in S204 includes: terminating the iteration when the length of the current candidate code sequence exceeds a length threshold or the number of completed sequences exceeds a quantity threshold.

[0024] Specifically, calculating the candidate code sequence score of the current candidate code sequence or the completed sequence includes:

[0025] Syntax and lexical analysis are performed on the current candidate code sequence or the completed sequence to build an abstract syntax tree. The abstract syntax tree is converted into a text vector and the text semantic features of the text vector are extracted. The similarity between the text semantic features of the text vector and the requirement semantic vector of the multi-dimensional requirement prompt words is calculated to obtain the functional score of the candidate code sequence.

[0026] A node relationship graph is generated based on the abstract syntax tree. Path features of the node relationship graph are extracted and input into a pre-trained candidate code sequence structure scoring model to obtain a candidate code sequence structure score. The path features include the average depth of the complete path, the average depth of the incomplete path, the average depth of the overall path, the variance of the overall path depth, and the ratio of the average depth of the incomplete path to the average depth of the overall path.

[0027] Set a functional weight factor and a structural weight factor, and calculate the product of the functional weight factor and the functional score of the candidate code sequence and the product of the structural weight factor and the structural score of the candidate code sequence to obtain the score of the candidate code sequence.

[0028] Specifically, calculating the candidate code sequence score of the current candidate code sequence or the completed sequence further includes:

[0029] Obtain the multi-indicator scores of the current candidate code sequence or the completed sequence and set the corresponding indicator weight factors. The multi-indicator scores include compilation pass rate index score, functional correctness index score, execution efficiency index score, code readability and maintainability index score, and generation speed index score.

[0030] Calculating the candidate code sequence score of the current candidate code sequence specifically includes: calculating the average of the products of the compilation pass rate index score, the functional correctness index score, the execution efficiency index score, and the code readability and maintainability index score multiplied by their respective index weight factors, to obtain the candidate code sequence score of the current candidate code sequence;

[0031] Calculating the candidate code sequence score of the completed sequence specifically includes: calculating the average of the products of the multi-index scores multiplied by the corresponding index weight factors to obtain the candidate code sequence score of the completed sequence.

[0032] Specifically, it also includes S4, which converts the code output after iterative optimization of the large language model into Java source code, then uses the JavaCompiler API to compile the Java source code into bytecode files, and finally uses the JarOutputStream class to package the bytecode files into a JAR file. The principle of dynamically building UDF dependency JAR files in Java lies in using Java reflection mechanism and bytecode manipulation technology to generate and compile UDF code at runtime, and finally package it into an executable JAR file for direct calling in big data processing frameworks such as Flink SQL.

[0033] According to one aspect of the present invention, a system for dynamically constructing UDFs based on large models is proposed, comprising the following modules:

[0034] The prompt word construction module is configured to construct multi-dimensional requirement prompt words for user-defined functions based on the characteristics and usage requirements of the target platform. The multi-dimensional requirement prompt words include function description prompt words, input / output requirement prompt words, and auxiliary information prompt words.

[0035] The UDF code generation module is configured to input the multi-dimensional requirement prompts into a pre-trained large language model to generate code. The large language model performs multiple iterations based on the multi-dimensional requirement prompts and outputs the optimal code sequence that implements the user-defined function.

[0036] The post-processing optimization module is configured to perform code style standardization on the optimal code sequence, and call the compiler or interpreter and static analysis tool to perform syntax checking and static analysis on the optimal code sequence respectively. Based on the error logs output by the compiler or interpreter and the static analysis results output by the static analysis tool, optimization prompt words are constructed, and the optimization prompt words are input into the large language model to iteratively optimize the optimal code sequence.

[0037] According to one aspect of the present invention, a computer program product is provided having a computer program stored thereon, which, when executed by a processor, performs the method as described in the first aspect.

[0038] The advantages of this invention lie in its utilization of the strengths of large-scale intelligent agents in natural language understanding and code generation. By designing appropriate prompts and optimizing the generation process, it achieves automated UDF generation. Research on dynamic UDF generation based on large models aims to address the pain points of traditional UDF writing methods by combining the powerful learning and generation capabilities of large models. From a theoretical perspective, this method explores the deep integration of artificial intelligence and big data technologies, providing a new technological paradigm for the data processing field. From a practical perspective, this method significantly lowers the programming threshold, allowing non-professional users to easily participate in the UDF generation process, thereby improving the accessibility and popularity of big data processing technologies. Simultaneously, the dynamic generation method based on large models can greatly improve development efficiency, reduce manual intervention, and ensure the accuracy, reliability, and performance optimization of the generated UDF code. Therefore, the research and implementation of this method not only has significant academic value but also profound implications for promoting the practical application of big data technologies across various industries. Attached Figure Description

[0039] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0040] Figure 1 A flowchart illustrating a method for dynamically constructing a UDF based on a large model according to the present invention is shown.

[0041] Figure 2 A schematic diagram of a system for dynamically constructing UDFs based on a large model according to the present invention is shown;

[0042] Figure 3 A schematic diagram of a computer system architecture suitable for implementing the embodiments of this application is shown. Detailed Implementation

[0043] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0044] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0045] Figure 1 According to one aspect of the present invention, a method for dynamically constructing a UDF based on a large model is proposed, comprising the following steps:

[0046] S1. Construct multi-dimensional requirement prompts for user-defined functions based on the characteristics and usage requirements of the target platform. The multi-dimensional requirement prompts include function description prompts, input / output requirement prompts, and auxiliary information prompts.

[0047] S2, the multi-dimensional requirement prompts are input into a pre-trained large language model to generate code. The large language model performs multiple iterations based on the multi-dimensional requirement prompts and outputs the optimal code sequence to implement the user-defined function.

[0048] S3, perform code style standardization on the optimal code sequence, and call the compiler or interpreter and static analysis tool to perform syntax checking and static analysis on the optimal code sequence respectively. Construct optimization prompt words based on the error logs output by the compiler or interpreter and the static analysis results output by the static analysis tool, and input the optimization prompt words into the large language model to iteratively optimize the optimal code sequence.

[0049] The structure of prompts directly affects the agent's understanding of the task and the accuracy of code generation. A typical prompt usually includes three parts: a functional description, input / output requirements, and other auxiliary information. Through this structured design, prompts can provide the agent with clear task guidance, thereby improving the accuracy and efficiency of code generation.

[0050] Specifically, S1 includes:

[0051] Based on the usage requirements, determine the function type of the user-defined function and construct corresponding function description prompts;

[0052] Based on the characteristics of the target platform, obtain the data types and language types supported by the target platform, determine the input parameter types and output result data types of the user-defined function based on the data types and language types supported by the target platform, and construct the input / output requirement prompt words;

[0053] Based on the characteristics and usage requirements of the target platform, determine whether the user-defined function needs to support parallel computing, whether a performance optimization threshold needs to be set, whether it has dependency library restrictions, whether it has a caching strategy, as well as the compatibility with the platform version and the requirements for exception handling, and construct the auxiliary information prompt words.

[0054] Designing effective prompts is crucial for guiding an agent to generate correct UDFs. The clarity, completeness, and relevance of the prompts directly impact the quality of the model's output. Clarity requires the prompts to be concise and unambiguous, ensuring the model accurately understands the user's intent. Completeness emphasizes that the prompts should include all necessary information, such as functional descriptions and input / output requirements, to reduce the likelihood of the model generating incorrect codes. Relevance means that the prompts need to be customized according to the specific task, ensuring the model focuses on the core requirements.

[0055] The structure of prompts directly impacts the agent's understanding of the task and the accuracy of code generation. A typical prompt usually includes three parts: a functional description, input / output requirements, and other auxiliary information. The functional description briefly explains the core functionality of the UDF, such as "implement a function to calculate the length of a string" or "design a table-valued function to filter duplicate elements." The input / output requirements define the parameter types and return format of the function in detail, such as "input is a string, output is an integer" or "input is a data stream, output is a filtered data stream." Furthermore, the auxiliary information section can contain additional constraints or optimization suggestions, such as the function needing to support parallel computing, prioritizing performance optimization, limiting dependencies, ensuring platform version compatibility, implementing caching strategies, and specifying exception handling guidelines. Through this structured design, prompts provide the agent with clear task guidance, thereby improving the accuracy and efficiency of code generation.

[0056] In particular, when generating UDFs, compared to generating general-purpose programming languages ​​(such as Java), prompt words need to incorporate more constraints and contextual information strongly related to specific computing engines or database systems. This is because UDFs do not run independently, but are deeply embedded in the execution framework of the host system (such as Flink, Spark, Hive, etc.), and their signatures, lifecycles, state management, and data interaction methods are strictly limited.

[0057] The following are optimization methods for prompt words generated by UDFs:

[0058] Explicitly specify the host platform and language: The target platform must be clearly specified at the beginning of the prompt (e.g., "Write a Java UDF for Apache Flink 1.18" or "Write a SQL UDF for PostgreSQL 15"), because the APIs, data type systems and function interfaces of different platforms vary greatly.

[0059] Precisely define the function type: The function type of a UDF is its core architecture and directly determines the code structure. Specifically, this includes:

[0060] Scalar functions: These functions take one or more fields from a row of data as input and return a single scalar value. Examples include string manipulation, mathematical calculations, and date conversions.

[0061] Table functions: These take a single row of data as input and can return zero, one, or multiple rows of data, each of which can contain multiple fields. For example, they can parse a JSON string into multiple rows of records.

[0062] Aggregate functions: These take a set of rows (within a group) of data as input, perform cumulative calculations, and return a single aggregate value. Examples include: summation, average, and custom window statistics.

[0063] Window Function (supported on some platforms): Performs calculations within a specific window area, typically requiring access to data in the current row and several rows before and after it.

[0064] A detailed description of the required data type is required: The data type must be precise to the specific type supported by the host platform, and cannot be vague. Specifically, this includes:

[0065] Input parameter data types: For example, in Flink, they should be explicitly String, Integer, Row, ArrayData, etc.; in databases, they should be explicitly VARCHAR(255), INT, TIMESTAMP, JSONB, etc.

[0066] The data type of the output result also needs to be specified precisely. For table-valued functions, the complete schema of the output table (field names and corresponding data types) also needs to be defined.

[0067] In some embodiments, an intermediate state data type (for aggregate functions) is also included: If it is an aggregate function, the data structure and type of the accumulator need to be explicitly defined, which is crucial for the correct implementation of the aggregate function.

[0068] In some embodiments, specific input and output examples are also provided. By providing one or two specific input examples and their corresponding expected outputs, the model can be greatly helped to understand the semantics and boundary conditions of the function.

[0069] Helper prompts declare necessary dependencies and context. If a function needs to access external resources (such as configuration files or dictionaries) or use specific libraries, this should be stated in the prompt. For example, when generating UDFs for Flink SQL, the prompt should not simply ask "Write a UDF," but should be optimized to: "Please write a Java scalar function named CalculateDiscount for Apache Flink 1.18. It accepts two parameters: the first is the original price of the product (type DECIMAL(10,2)), and the second is the membership level (type STRING, with values ​​'Gold', 'Silver', 'Bronze'). The function returns the discounted price (type DECIMAL(10,2)) based on the membership level, with the following rules: Gold gets an 80% discount, Silver gets a 90% discount, and Bronze gets no discount. Please provide a complete class definition and core calculation logic." Research shows that prompts that follow these principles and are specifically optimized can significantly improve the accuracy, usability, and compatibility with the host system in UDF code generation tasks.

[0070] Key technologies for dynamically generating UDFs mainly include model fine-tuning and code generation optimization algorithms. Model fine-tuning refers to retraining a pre-trained model using domain-specific data (such as a large number of Flink SQL UDF examples, API documentation, and best practices) to improve the model's performance on the target task. For example, by introducing Flink SQL-related code samples, the model can become more familiar with the framework's syntax, function structure, and context dependencies, thereby generating syntactically correct, logically coherent UDF code that conforms to the framework's specifications.

[0071] When selecting a large model as the foundation for an agent, its comprehensive performance in both natural language processing and code generation tasks must be considered. In recent years, the Qwen series of large language models (LLMs) has gained widespread attention for its superior text generation capabilities and deep contextual understanding. Qwen3, as an advanced large language model, demonstrates significant advantages in code generation. Through pre-training on massive datasets, it accurately understands and captures complex semantic relationships and syntactic structures, resulting in code that is not only highly accurate but also highly readable and structured. Compared to other types of models, Qwen3 exhibits greater flexibility and adaptability in handling long-sequence text generation tasks, making it particularly suitable for scenarios requiring the dynamic generation of diverse and complex code snippets. Furthermore, Qwen3's powerful capabilities in natural language understanding and code logic grasp allow it to better customize and generate context-aware code according to developers' needs, significantly improving development efficiency and code quality. Therefore, this method selects Qwen3 as the core model for the agent. This model not only performs strongly in natural language processing but also demonstrates excellent performance in code generation, laying a solid foundation for its application in dynamic UDF generation.

[0072] Specifically, the large language model described in S2, based on the multi-dimensional requirement prompt words, undergoes multiple rounds of iteration, including:

[0073] S201, the large language model generates an initial code sequence based on the multi-dimensional requirement prompt words;

[0074] S202, the large language model generates multiple candidate code sequences and corresponding candidate code sequence scores based on the initial code sequence, and retains a predetermined number of candidate code sequences as the current candidate code sequences based on the candidate code sequence scores;

[0075] S203, the large language model generates a new candidate code sequence based on the current candidate code sequence and updates the corresponding candidate code sequence score, and retains a predetermined number of candidate code sequences as new current candidate code sequences based on the candidate code sequence score;

[0076] S204, the large language model repeats S203 for multiple rounds of iteration until the termination condition is met;

[0077] S205, after the current candidate code sequence generates an end marker, it is treated as a completed sequence. The candidate code sequence scores of the completed sequences that meet the termination conditions are obtained, processed, and filtered, and the optimal code sequence is output.

[0078] During the process of generating code word by word (or token by token), the model maintains multiple candidate sequences in parallel. At each step, it retains several partial sequences with the highest probability, and finally selects the one with the highest comprehensive score from these complete sequences as the output. This avoids local optima and can explore code paths with more complete semantics and more reasonable structure.

[0079] Specifically, satisfying the termination condition in S204 includes: terminating the iteration when the length of the current candidate code sequence exceeds a length threshold or the number of completed sequences exceeds a quantity threshold.

[0080] Specifically, calculating the candidate code sequence score of the current candidate code sequence or the completed sequence includes:

[0081] Syntax and lexical analysis are performed on the current candidate code sequence or the completed sequence to build an abstract syntax tree. The abstract syntax tree is converted into a text vector and the text semantic features of the text vector are extracted. The similarity between the text semantic features of the text vector and the requirement semantic vector of the multi-dimensional requirement prompt words is calculated to obtain the functional score of the candidate code sequence.

[0082] A node relationship graph is generated based on the abstract syntax tree. Path features of the node relationship graph are extracted and input into a pre-trained candidate code sequence structure scoring model to obtain a candidate code sequence structure score. The path features include the average depth of the complete path, the average depth of the incomplete path, the average depth of the overall path, the variance of the overall path depth, and the ratio of the average depth of the incomplete path to the average depth of the overall path.

[0083] Set a functional weight factor and a structural weight factor, and calculate the product of the functional weight factor and the functional score of the candidate code sequence and the product of the structural weight factor and the structural score of the candidate code sequence to obtain the score of the candidate code sequence.

[0084] In one embodiment, structuring the code into a tree structure using an Abstract Syntax Tree (AST) and then converting it into text vectors captures the semantic information of the code. By calculating the similarity between the text vectors of the code and the semantic vectors of the multi-dimensional requirement prompts, it can be ensured that the generated code functionally meets user needs. By extracting path features from the AST (such as average depth and depth variance), the structural characteristics of the code can be quantified. These features reflect the complexity and nesting depth of the code, which are related to its readability and maintainability. Evaluating the code from both functional compliance and structural quality dimensions allows the scoring to consider both semantics and structure. Since the score consists of two interpretable parts, we can analyze the functional score and the structural score separately to understand where the code is lacking.

[0085] Specifically, calculating the candidate code sequence score of the current candidate code sequence or the completed sequence further includes:

[0086] Obtain the multi-indicator scores of the current candidate code sequence or the completed sequence and set the corresponding indicator weight factors. The multi-indicator scores include compilation pass rate index score, functional correctness index score, execution efficiency index score, code readability and maintainability index score, and generation speed index score.

[0087] Calculating the candidate code sequence score of the current candidate code sequence specifically includes: calculating the average of the products of the compilation pass rate index score, the functional correctness index score, the execution efficiency index score, and the code readability and maintainability index score multiplied by their respective index weight factors, to obtain the candidate code sequence score of the current candidate code sequence;

[0088] Calculating the candidate code sequence score of the completed sequence specifically includes: calculating the average of the products of the multi-index scores multiplied by the corresponding index weight factors to obtain the candidate code sequence score of the completed sequence.

[0089] In one embodiment, the quality and efficiency of the generated code are reflected by the following core metrics:

[0090] 1. Compilation Success Rate: Measures whether the generated code is syntactically correct and can be successfully compiled by the target environment (such as Flink).

[0091] 2. Functional Correctness: Verify whether the generated UDF can produce the expected output results under a given input through unit testing or integration testing.

[0092] 3. Execution Efficiency: Compare the generated code with manually written standard or benchmark code on the same dataset in terms of runtime, CPU, and memory consumption.

[0093] 4. Code readability and maintainability: This is evaluated through metrics such as code complexity (e.g., cyclomatic complexity), comment coverage, and compliance with team coding standards. It usually requires manual review or the assistance of static code analysis tools.

[0094] 5. Generation Latency: The time taken from receiving the requirements to outputting the final optimized code, which is crucial for dynamic generation scenarios.

[0095] By setting threshold conditions, the various indicators of the current candidate code sequence or the completed sequence are scored, and corresponding weight factors are set to adapt to the needs of specific application scenarios. Finally, the average value of the product of each indicator and its corresponding weight is calculated to obtain the score of the candidate code sequence, thereby achieving the selection of the optimal code sequence.

[0096] The process of dynamically generating UDFs based on a large-model agent can be divided into three main steps: prompt input, code generation, and post-processing. First, the user writes prompts according to task requirements and inputs them into the agent. Then, the agent uses a pre-trained model to reason about the prompts, extracting key information and generating a preliminary code framework. Next, the model gradually fills in the code details based on the functional descriptions and input / output requirements in the prompts, ultimately generating the complete UDF code. Finally, in the post-processing stage, the system does not simply perform validation but initiates a closed-loop "compile-diagnose-repair-optimize" workflow to ensure the usability and robustness of the generated code.

[0097] After obtaining the initially generated code, a rule-based optimization strategy is introduced for secondary refinement. This includes:

[0098] 1. Code style guidelines: Automatic code formatting, consistent indentation, naming conventions (such as camelCase), and comment style to improve readability and maintainability.

[0099] 2. Performance Optimization Recommendations: Static analysis of the generated code identifies potential performance bottlenecks. For example, it may detect and suggest moving invariant calculations from inside loops to outside loops, recommend using more efficient data structures (such as using HashSet instead of ArrayList for member checking), or suggest avoiding unnecessary object creation.

[0100] 3. Syntax and semantic verification: The generated code is quickly verified by calling the compiler or interpreter's API to ensure that its syntax is correct and that it passes basic type checks.

[0101] During syntax checking and static analysis, the system invokes the target language's actual compiler (such as javac, scalac) or interpreter (such as the Python interpreter) to fully compile or parse the generated source code, capturing all compile-time errors such as syntax errors, type mismatches, undeclared variables, and inconsistent method signatures. Static code analysis tools (through integration with tools such as Checkstyle (Java), Pylint (Python), and ESLint (JavaScript)) check code style, potential logical errors (such as null pointer risks and unclosed resources), and compliance with the host platform's (such as Flink and Spark) coding standards.

[0102] The system automatically parses the error logs output by the compiler or interpreter and analysis tools, extracting key information: error type, error line number, error description, and contextual code snippets. Using structured logs or natural language processing techniques, it transforms machine error messages into more easily understandable "diagnostic reports," such as: "Line 25: The parameter type for method 'accumulate' should be 'MyAccumulator', but 'String' was actually passed in."

[0103] The system then re-inputs the diagnostic report as "negative feedback" to the large language model, prompting it: "The original code has a [specific error type] on line X. Please correct and regenerate that part of the code." The model then performs a partial rewrite based on the context and error information.

[0104] If the initial fix still fails to compile, the system will perform multiple iterations (usually with a maximum number of retries, such as 3). Each time, the latest error report and the code snippet generated in the previous round will be used as context input to guide the model to gradually approach the correct version.

[0105] For high-frequency, patterned errors (such as missing override annotations or incorrect constructor signatures), the system can have built-in repair templates or rule engines for automated replacement, reducing reliance on large models and improving repair efficiency.

[0106] Through this series of automated and intelligent post-processing mechanisms, the system not only ensures that the generated code "can run," but also moves towards production-grade standards of "running correctly, running fast, and being easy to maintain." It verifies whether the generated UDF correctly implements the interfaces required by the host platform (such as Flink's ScalarFunction and AggregateFunction), whether the method name, parameter list, and return type strictly match, enhances code readability, and injects performance suggestions.

[0107] In an internal experiment, two methods were compared: "pure model generation" (using only the fine-tuned model without optimization algorithms) and "model + optimization algorithm generation".

[0108] Unoptimized code: While the code directly output by the model is generally functionally correct (approximately 75% accuracy), it suffers from numerous inconsistencies in style, arbitrary variable naming, and redundant loops that could be optimized. Its compilation success rate is only 80%, and its average execution time is 40% slower than the benchmark code.

[0109] The optimized code achieved a 98% compilation pass rate and a 92% functional accuracy rate. More importantly, after performance rule optimization, its average execution efficiency was almost identical to the manually written baseline code (only within 5% slower), and the code style fully complied with team standards, greatly reducing the cost of subsequent manual review and modification.

[0110] Specifically, it also includes S4, which converts the code output after iterative optimization of the large language model into Java source code, then uses the JavaCompiler API to compile the Java source code into bytecode files, and finally uses the JarOutputStream class to package the bytecode files into a jar package.

[0111] The principle behind dynamically building UDF dependency JAR files in Java lies in leveraging Java reflection and bytecode manipulation techniques to generate and compile UDF code at runtime, ultimately packaging it into an executable JAR file. Compared to traditional compilation methods, dynamic building eliminates the need to write complete source code beforehand; instead, it automatically generates code and completes the compilation process based on user-input prompts. This approach not only simplifies the development process but also improves code flexibility and adaptability. Specifically, the dynamic building process first generates Java source code through string concatenation, then compiles it into bytecode files using the JavaCompiler API. Finally, the JarOutputStream class is used to package the compiled bytecode files into a JAR file for direct use in big data processing frameworks such as Flink SQL.

[0112] The following code example details how to dynamically build and compile UDF dependency JAR files using Java. First, define a string variable `code` to store the generated Java source code. For example, the source code for a UDF that calculates string length might look like this:

[0113] public class StringLengthUDF{

[0114] public static int calculateLength(String input){

[0115] return input.length();

[0116] }

[0117] }

[0118] Next, use the JavaCompiler class to compile the code:

[0119] JavaCompiler compiler=ToolProvider.getSystemJavaCompiler();

[0120] StandardJavaFileManager fileManager=

[0121] compiler.getStandardFileManager(null,null,null);

[0122] Iterable<? extends JavaFileObject>compilationUnits=Arrays.asList(newStringJavaObject(code));

[0123] CompilationTask task=compiler.getTask(null,fileManager,null,null,

[0124] null,compilationUnits);

[0125] boolean success=task.call();

[0126] fileManager.close();

[0127] After successful compilation, use JarOutputStream to package the generated bytecode files into a JAR file:

[0128] JarOutputStream jarOut=new JarOutputStream(newFileOutputStream("StringLengthUDF.jar"));

[0129] jarOut.putNextEntry(new JarEntry("StringLengthUDF.class"));

[0130] jarOut.write(bytecode); / / bytecode is the array of bytecode generated during compilation jarOut.close();

[0131] By following the steps above, the dynamic construction of UDF dependency JAR files can be completed, thereby achieving seamless integration into big data processing frameworks such as FlinkSQL.

[0132] According to one aspect of the invention, such as Figure 2 As shown, a system for dynamically constructing UDFs based on a large model is proposed, including the following modules:

[0133] The prompt word construction module 201 is configured to construct multi-dimensional requirement prompt words for user-defined functions based on the characteristics and usage requirements of the target platform. The multi-dimensional requirement prompt words include function description prompt words, input / output requirement prompt words, and auxiliary information prompt words.

[0134] The UDF code generation module 202 is configured to input the multi-dimensional requirement prompts into a pre-trained large language model for code generation. The large language model performs multiple iterations based on the multi-dimensional requirement prompts and outputs the optimal code sequence that implements the user-defined function.

[0135] The post-processing optimization module 203 is configured to perform code style standardization on the optimal code sequence, and call the compiler or interpreter and static analysis tool to perform syntax checking and static analysis on the optimal code sequence respectively. Based on the error logs output by the compiler or interpreter and the static analysis results output by the static analysis tool, optimization prompt words are constructed, and the optimization prompt words are input into the large language model to iteratively optimize the optimal code sequence.

[0136] The following is for reference. Figure 3 It shows a schematic diagram of the structure of a computer system 300 suitable for implementing electronic devices according to embodiments of the present application. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0137] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which performs various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 309 into random access memory (RAM) 304. The RAM 304 also stores various programs and data required for the operation of the system 300. The CPU 301, ROM 302, ROM 303, and RAM 304 are interconnected via a bus 305. An input / output (I / O) interface 306 is also connected to the bus 305.

[0138] The following components are connected to I / O interface 306: an input section 307 including a keyboard, mouse, etc.; an output section 308 including a liquid crystal display (LCD) and speakers, etc.; a storage section 309 including a hard disk, etc.; and a communication section 310 including a network interface card such as a LAN card and a modem, etc. The communication section 310 performs communication processing via a network such as the Internet. A drive 311 is also connected to I / O interface 306 as needed. A removable medium 312, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 311 as needed so that computer programs read from it can be installed into storage section 309 as needed.

[0139] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts are implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program is downloaded and installed from a network via communication section 310, and / or installed from removable medium 312. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the methods of this application.

[0140] It should be noted that the computer-readable storage medium of this application is a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium is, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium is any tangible medium containing or storing a program that is used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium includes a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals take various forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium or any computer-readable storage medium other than a computer-readable storage medium may transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0141] Computer program code for performing the operations of this application is written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code executes entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer is connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram represents a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually execute substantially in parallel, and they may sometimes execute in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, is implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0143] The modules described in the embodiments of this application are implemented in software or hardware.

[0144] In another aspect, this application also provides a computer-readable storage medium included in the electronic device described in the above embodiments; or existing independently and not assembled into the electronic device. The computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform: S1, constructing multi-dimensional requirement prompts for user-defined functions based on the characteristics and usage requirements of the target platform, the multi-dimensional requirement prompts including functional description prompts, input / output requirement prompts, and auxiliary information prompts; S2, inputting the multi-dimensional requirement prompts into a pre-trained large language model for code generation, the large language model performing multiple iterations based on the multi-dimensional requirement prompts to output the optimal code sequence for implementing the user-defined function; S3, performing code style standardization on the optimal code sequence, and respectively calling a compiler or interpreter and a static analysis tool to perform syntax checking and static analysis on the optimal code sequence, constructing optimization prompts based on the error logs output by the compiler or interpreter and the static analysis results from the static analysis tool, and inputting the optimization prompts into the large language model to iteratively optimize the optimal code sequence.

[0145] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for dynamically constructing UDFs based on large models, characterized in that, Includes the following steps: S1. Construct multi-dimensional requirement prompts for user-defined functions based on the characteristics and usage requirements of the target platform. The multi-dimensional requirement prompts include function description prompts, input / output requirement prompts, and auxiliary information prompts. S2, the multi-dimensional requirement prompts are input into a pre-trained large language model to generate code. The large language model performs multiple iterations based on the multi-dimensional requirement prompts and outputs the optimal code sequence to implement the user-defined function. S3, perform code style standardization on the optimal code sequence, and call the compiler or interpreter and static analysis tool to perform syntax checking and static analysis on the optimal code sequence respectively. Construct optimization prompt words based on the error logs output by the compiler or interpreter and the static analysis results output by the static analysis tool, and input the optimization prompt words into the large language model to iteratively optimize the optimal code sequence.

2. The method for dynamically constructing a UDF based on a large model according to claim 1, characterized in that, S1 specifically includes: Based on the usage requirements, determine the function type of the user-defined function and construct corresponding function description prompts; Based on the characteristics of the target platform, obtain the data types and language types supported by the target platform, determine the input parameter types and output result data types of the user-defined function based on the data types and language types supported by the target platform, and construct the input / output requirement prompt words; Based on the characteristics and usage requirements of the target platform, determine whether the user-defined function needs to support parallel computing, whether a performance optimization threshold needs to be set, whether it has dependency library restrictions, whether it has a caching strategy, as well as the compatibility with the platform version and the requirements for exception handling, and construct the auxiliary information prompt words.

3. The method for dynamically constructing a UDF based on a large model according to claim 2, characterized in that, The user-defined functions include scalar functions, table-valued functions, aggregate functions, and window functions.

4. The method for dynamically constructing a UDF based on a large model according to claim 1, characterized in that, The large language model described in S2 undergoes multiple iterations based on the multi-dimensional requirement prompt words, specifically including: S201, the large language model generates an initial code sequence based on the multi-dimensional requirement prompt words; S202, the large language model generates multiple candidate code sequences and corresponding candidate code sequence scores based on the initial code sequence, and retains a predetermined number of candidate code sequences as the current candidate code sequences based on the candidate code sequence scores; S203, the large language model generates a new candidate code sequence based on the current candidate code sequence and updates the corresponding candidate code sequence score, and retains a predetermined number of candidate code sequences as new current candidate code sequences based on the candidate code sequence score; S204, the large language model repeats S203 for multiple rounds of iteration until the termination condition is met; S205, after the current candidate code sequence generates an end marker, it is treated as a completed sequence. The candidate code sequence scores of the completed sequences that meet the termination conditions are obtained, processed, and filtered, and the optimal code sequence is output.

5. The method for dynamically constructing a UDF based on a large model according to claim 4, characterized in that, Specifically, satisfying the termination condition in S204 includes: terminating the iteration when the length of the current candidate code sequence exceeds a length threshold or the number of completed sequences exceeds a quantity threshold.

6. The method for dynamically constructing a UDF based on a large model according to claim 4, characterized in that, Calculating the candidate code sequence score of the current candidate code sequence or the completed sequence specifically includes: Syntax and lexical analysis are performed on the current candidate code sequence or the completed sequence to build an abstract syntax tree. The abstract syntax tree is converted into a text vector and the text semantic features of the text vector are extracted. The similarity between the text semantic features of the text vector and the requirement semantic vector of the multi-dimensional requirement prompt words is calculated to obtain the functional score of the candidate code sequence. A node relationship graph is generated based on the abstract syntax tree. Path features of the node relationship graph are extracted and input into a pre-trained candidate code sequence structure scoring model to obtain a candidate code sequence structure score. The path features include the average depth of the complete path, the average depth of the incomplete path, the average depth of the overall path, the variance of the overall path depth, and the ratio of the average depth of the incomplete path to the average depth of the overall path. Set a functional weight factor and a structural weight factor, and calculate the product of the functional weight factor and the functional score of the candidate code sequence and the product of the structural weight factor and the structural score of the candidate code sequence to obtain the score of the candidate code sequence.

7. The method for dynamically constructing a UDF based on a large model according to claim 4, characterized in that, Calculating the candidate code sequence score of the current candidate code sequence or the completed sequence further includes: Obtain the multi-indicator scores of the current candidate code sequence or the completed sequence and set the corresponding indicator weight factors. The multi-indicator scores include compilation pass rate index score, functional correctness index score, execution efficiency index score, code readability and maintainability index score, and generation speed index score. Calculating the candidate code sequence score of the current candidate code sequence specifically includes: calculating the average of the products of the compilation pass rate index score, the functional correctness index score, the execution efficiency index score, and the code readability and maintainability index score multiplied by their respective index weight factors, to obtain the candidate code sequence score of the current candidate code sequence; Calculating the candidate code sequence score of the completed sequence specifically includes: calculating the average of the products of the multi-index scores multiplied by the corresponding index weight factors to obtain the candidate code sequence score of the completed sequence.

8. The method for dynamically constructing a UDF based on a large model according to claim 1, characterized in that, It also includes S4, which converts the code output after iterative optimization of the large language model into Java source code, then uses the JavaCompiler API to compile the Java source code into bytecode files, and finally uses the JarOutputStream class to package the bytecode files into a jar package.

9. A system for dynamically constructing UDFs based on large models, characterized in that, Includes the following modules: The prompt word construction module is configured to construct multi-dimensional requirement prompt words for user-defined functions based on the characteristics and usage requirements of the target platform. The multi-dimensional requirement prompt words include function description prompt words, input / output requirement prompt words, and auxiliary information prompt words. The UDF code generation module is configured to input the multi-dimensional requirement prompts into a pre-trained large language model to generate code. The large language model performs multiple iterations based on the multi-dimensional requirement prompts and outputs the optimal code sequence that implements the user-defined function. The post-processing optimization module is configured to perform code style standardization on the optimal code sequence, and call the compiler or interpreter and static analysis tool to perform syntax checking and static analysis on the optimal code sequence respectively. Based on the error logs output by the compiler or interpreter and the static analysis results output by the static analysis tool, optimization prompt words are constructed, and the optimization prompt words are input into the large language model to iteratively optimize the optimal code sequence.

10. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

Images