A hybrid front-end framework migration method based on AST and LLM

Abstract

The application discloses a hybrid front-end framework migration method based on AST and LLM, and comprises the following steps: (1) original front-end framework project loading, identifying project files and dividing functional project code into logical processing code and user interface definition code; (2) lexical analysis, syntax analysis and semantic extraction; (3) code rewriter processing, generating intermediate code; (4) increment training optimization generates customized large model LLM, and the code function description of the original project file is obtained by using the customized large model LLM, and the target framework code is further generated; (5) based on the intermediate code, a syntax error report and processing suggestions are generated by the customized large model LLM; the intermediate code is optimized by using the target framework code, and the final migration code is generated. By using the application, the migration accuracy can be improved, and the syntax error report and processing suggestions can be given.

Description

Technical Field

[0001] This invention belongs to the field of code migration, and in particular relates to a hybrid front-end framework migration method based on AST and LLM. Background Technology

[0002] In the client-side systems of financial institutions, most use a UI framework based on Microsoft WPF. WPF is tightly coupled with the Windows operating system and cannot support domestically developed Linux-based operating systems. Currently, there is still no solution to the compatibility issues of the WPF framework with other operating systems.

[0003] Migrating desktop financial software to a domestically developed operating system is essentially a complex and comprehensive software conversion project. The complexity stems from the fact that it is almost equivalent to refactoring the entire software system, inevitably leading to a significant investment of human resources and time.

[0004] More and more companies are attempting to automate code migration. For example, domestic patent document CN116204208A proposes a method and apparatus for migrating code to upgrade front-end frameworks, and patent document CN115951890A proposes a method, system, and apparatus for code conversion between different front-end frameworks. Automating this migration process can not only reduce the investment of human resources but also significantly reduce the time required for the entire migration process.

[0005] Large Language Models (LLMs), such as GPT-3.5 or similar models, have made significant progress in the field of natural language processing, capable of generating text, answering questions, and engaging in dialogue. Large language models hold great potential for code generation and can be used for tasks such as automatic code generation and programming assistance. Therefore, introducing LLMs into desktop software migration is feasible.

[0006] However, the process of generating code using large language models has some shortcomings:

[0007] 1) Due to the large amount of code in mature software projects, and the content limitations of LLM's interface token (the smallest unit of text that can be processed), in many cases we cannot directly upload the complete project file content to LLM.

[0008] 2) As front-end framework versions are constantly updated, the content of various control libraries and APIs will also change. LLM is based on pre-training on historical data, and the information it generates has a certain time sensitivity. It cannot generate highly accurate content based on the latest data, which may result in some inaccuracies or inapplicability of the generated code snippets or suggestions when dealing with the latest front-end development needs.

[0009] 3) Unlike generating general text, simply processing uploaded code snippets is insufficient for accurately generating migration code. This is because the structure and logic of the code require a higher level of contextual understanding to ensure that the generated migration code remains consistent in both semantics and functionality.

[0010] 4) In domain-specific information processing, the training data for large language models (LLMs) may not cover information such as the company's internal development processes, frameworks, and content. To obtain accurate feedback, it is necessary to supplement this domain-specific content and relevant context. Summary of the Invention

[0011] This invention provides a hybrid front-end framework migration method based on AST and LLM, which can improve migration accuracy and also provide syntax error reports and processing suggestions.

[0012] A hybrid front-end framework migration method based on AST and LLM includes the following steps:

[0013] (1) Load the original front-end framework project, identify the project files, and divide the functional project code into logic processing code and user interface definition code;

[0014] (2) Lexical analysis, syntactic analysis and semantic extraction, the specific steps are as follows:

[0015] (2-1) Lexical analysis, which decomposes the logic processing code and user interface definition code into a series of lexical units;

[0016] (2-2) Syntax analysis: Based on lexical analysis, syntax analysis is performed to organize lexical units into an abstract syntax tree (AST) to represent the hierarchical structure of the code and the nesting relationships between code elements.

[0017] (2-3) Semantic analysis: Semantic analysis is performed on the basis of AST, including type checking, scope analysis, symbol resolution, inheritance and implementation analysis, etc.

[0018] (3) Code rewriter processing, the specific steps are as follows:

[0019] (3-1) Based on the control library of the original front-end framework and the target framework, write the implementation code of each method and property based on the control library;

[0020] (3-2) Based on the correspondence between the properties and method calls of each control before and after migration, the migration rules are summarized according to the complexity. The migration rules are divided into: single-point migration, single-statement migration, and complex structure migration.

[0021] (3-3) Implement a code rewriter based on the semantic analysis AST model and transfer rules to generate intermediate code;

[0022] (4) Incremental training and optimization to generate a customized large model LLM, using the customized large model LLM to obtain the code function description of the original project file, and further generate the target framework code;

[0023] (5) Based on the intermediate code, generate syntax error reports and processing suggestions through a customized large model LLM; optimize the intermediate code using the target framework code to generate the final migration code.

[0024] The code migration method of the present invention processes code according to its function, performs semantic analysis, constructs a semantic model, and merges the intermediate code generated by the code rewriter and the target framework code generated by the LLM model to generate the final migration code. This solves the problems of cumbersome and labor-intensive manual migration, automates the migration process, improves migration efficiency, and reduces workload and cost.

[0025] The specific process of step (1) is as follows:

[0026] Use code compilation analysis tools such as Roslyn to load the pre-compiled version of the entire project, identify the contents of the project files, classify the files into logic processing files, user interface definition files, resource files, configuration files, and other files, extract the logic processing code files and user interface definition code files, and generate the mapping relationship between the logic processing code files and user interface definition code files.

[0027] In step (2-1), the logic processing code is decomposed into keywords, identifiers, operators, constants, etc., and the user interface definition code is decomposed into label names, attribute names, and attribute values.

[0028] In steps (2-3), during the type checking phase, the data types of variables, expressions, and values ​​in the code will be determined; during the scope analysis phase, the scope of variables, functions, and classes will be identified, and the declaration and use of variables within the scope will be checked; during the symbol resolution phase, a symbol table will be built to store information about variables, functions, and classes, and symbols will be identified and declarations and references will be found; during the inheritance and implementation analysis phase, the inheritance relationships and interfaces between classes will be analyzed.

[0029] In step (3-2), for single-point migration, the migration is updated by traversing the syntax tree; for single-statement migration, the migration is achieved by matching the target statement with regular expressions and extracting the call information; for complex structure migration, the migration is achieved by modifying the syntax tree for each rule.

[0030] In step (3-3), the code rewriter is generated based on the AST operation tree nodes to modify, move, and reorganize the code; and it is divided into regular rule rewriter, special rule rewriter, and subsequent optimization rewriter based on migration rules.

[0031] This process is divided into several parts, including extracting the framework control library content before and after migration, generating migration rules, refactoring code, and handling related issues.

[0032] During the framework control library content extraction phase, a web crawler framework was used to crawl official documentation, open-source projects, and community Q&A information from both the source framework software library and the target migration software library. The crawled information was then processed. Simultaneously, to ensure that the migration rules corresponded one-to-one at the declaration space, control level, class level, method level, and statement body level, the methods and properties of each front-end control in the original framework were implemented to correspond with the methods and properties of the target migration framework.

[0033] During the migration rule generation phase, by comparing the crawled content with the control library code of the original framework and the target framework before and after migration, a one-by-one matching and mapping is performed based on coarse-grained levels such as project file level, control level, declaration space level, class level, and method body level, and a fine-grained mapping is performed based on statement body level and identifier level to generate migration rules based on statement level.

[0034] Specifically, we summarized three types of migration rules: (1) Single-point migration. This type of rule includes the migration of a single syntax node in the AST, such as modifying variable attribute names, function names, etc. For this type, we use a code rewriter to traverse the syntax tree for rewriting. (2) Single-statement migration: This type of migration involves the migration of a single statement. We use a code rewriter to traverse the code statement by statement and use regular expressions for matching and migration. (3) Complex structure migration. This type of migration is often highly variable and complex. We perform special processing for each migration. This is achieved by directly modifying the AST.

[0035] During the code refactoring phase, the semantic model generated by the original front-end framework project is loaded, and the statements to be migrated are located based on the AST nodes. The location process uses regular expression matching, code analysis tools to match nodes one by one, and other methods to generate the located statements into the target framework statements.

[0036] During the association processing phase, the declarations and calls of related variables, functions, and classes in the context are recursively searched based on the symbol table. For example, based on the declaration method of a class, the inheritance relationship of the class is searched to locate the first declaration of the method and modify the relevant content.

[0037] The specific process of step (4) is as follows:

[0038] (4-1) Load external data, relevant documents and migration rule summary information as data sources into the form of large model reading, divide the data into specified sizes and store them in the vector database in the form of embedding, pass them to the large model, and generate a customized large model LLM.

[0039] (4-2) Statistically analyze all code to be migrated in the project, determine whether the token limit has been exceeded, and split the original project files based on semantic analysis information;

[0040] (4-3) Based on the correspondence between the logic processing code and the user interface definition code, as well as the context call relationship, merge and upload the customized large model LLM to obtain the code function description of the original project file;

[0041] (4-4) Based on the functional description, call the customized large model LLM to generate the target framework code.

[0042] In step (4-1), the data source is divided into document blocks of a specified size and stored in the vector database in an embedded form. When inputting a prompt, the segmented document is retrieved from the vector database. By comparing the cosine similarity, the document block in the vector database that is similar to the prompt question is found. The document block is passed to the LLM model, and the prompt containing the question and the document block is used to generate the answer.

[0043] In step (5), the intermediate code is based on the content of the original framework code, and the target framework code is based on the functionality of the original framework code. The intermediate code is optimized using the target framework code. While maintaining the structure of the original code, the code rewriter optimizes the parts that do not conform to the syntax that it cannot handle, and generates error logs and optimization suggestions for subsequent developers to handle.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The method of this invention is based on the project architecture, syntax model, and actual call relationships. It parses the symbols and call information of the migration process at the granularity of abstract syntax tree nodes, statements, code blocks, namespaces, and project files, respectively, to achieve automated migration and obtain intermediate code. It also identifies the statements in the source code files based on the large model, parses their functions and relationships, optimizes the intermediate code based on this, and obtains the final migration code, improving the accuracy of migration. Furthermore, it provides test cases, marks special modification locations, and provides modification suggestions. Attached Figure Description

[0046] Figure 1 This is a flowchart of a hybrid front-end framework migration method based on AST and LLM according to the present invention.

[0047] Figure 2 This diagram illustrates the classification and processing methods of migration rules in this invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0049] like Figure 1 As shown, a hybrid front-end framework migration method based on AST and LLM mainly includes two aspects: processing of the original front-end project files and processing of external materials such as official documentation and open-source projects.

[0050] The processing of the original front-end project files mainly consists of three steps: S1 project file loading, S2 semantic analysis and model building, and S3 intermediate code generation. The processing of external materials such as official documentation and open-source projects mainly consists of S4 model customization, S5 code file processing, S6 chained calls, and S7 final code generation.

[0051] S1. Loading project files, the specific steps are as follows:

[0052] S11. Identify and read the project, analyze the project structure, and generate the project structure tree.

[0053] S12. Divide the project files into logic processing code files, user interface definition code files, and other files.

[0054] S2. Semantic analysis and model construction, the specific steps are as follows:

[0055] S21. Perform lexical and syntax analysis on the logic processing code file. Lexical analysis decomposes the source code text into a series of lexical units such as keywords, identifiers, and operators. Syntax analysis organizes the lexical units into an abstract syntax tree, identifying the hierarchical structure of the code and the nesting relationships between code elements. Semantic analysis is performed on the abstract syntax tree and includes type checking, scope analysis, member access, inheritance relationships, etc.

[0056] S22. Perform lexical and syntactic analysis on the user interface definition code. Lexical analysis decomposes the source code file into a series of lexical units such as tag names, attribute names, and attribute values. Syntactic analysis organizes element nodes, document nodes, attribute nodes, and text nodes into a document object model tree, and obtains the structure of the user interface definition through this model tree, so as to dynamically modify the content of the user interface definition.

[0057] S23. Based on the abstract syntax tree, semantic analysis is performed, executing a series of checks and transformations to ensure the semantic correctness of the source code. This includes type checking, scope analysis, constant folding, function and procedure call checking, type inference, reference resolution, exception handling, resource management, type system extension, and compile-time optimization for the source code.

[0058] In step S2, the processing of the logical processing file is divided into lexical analysis, syntax analysis, and semantic analysis. During the lexical analysis phase, the source code text is broken down into a series of lexical units, such as keywords, identifiers, operators, and constants. These lexical units are the basic building blocks of the code.

[0059] During the syntax analysis phase, lexical units are organized into an Abstract Syntax Tree (AST). The AST reflects the hierarchical structure and syntactic relationships of the code. Each node represents a code element, such as an expression, statement, function, or class.

[0060] In the semantic analysis phase, the focus is not only on the code structure but also on its meaning, performing type checking, scope analysis, symbol resolution, inheritance, and implementation analysis respectively. Type checking determines the data types of variables, expressions, and values ​​in the code; scope analysis identifies the scope of variables, functions, and classes, checking the declaration and use of variables within their scopes; the symbol resolution phase builds a symbol table to store information about variables, functions, and classes, identifying symbols and finding declarations and references; and the inheritance and implementation analysis phases analyze inheritance relationships and interfaces between classes.

[0061] In processing user interface definition files, these files are typically declarative markup languages ​​that contain information such as the hierarchical structure of user interface elements, styles, data binding, and interactions. This process includes lexical analysis, syntax analysis, element parsing, attribute parsing, event association, and style and template parsing.

[0062] During the lexical analysis phase, the source code text is broken down into a series of lexical units, such as element names, attribute names, and attribute values. During the syntax analysis phase, the lexical units are organized into a Document Object Model (DOM), which represents each element, attribute, text, and other content in the document as a tree structure.

[0063] During the element parsing phase, the name, type, and nesting relationship of each DOM node are identified. During the attribute parsing phase, the attribute name and value of each element's attributes are identified. During the data binding parsing phase, the source, target, and binding method of the binding are identified. During the event association phase, event handlers are identified, and events are associated with the processing methods in the logic code. During the style and template parsing phase, defined styles and templates need to be parsed into corresponding style and template objects. During the resource parsing phase, for elements referencing external resources, the reference source needs to be obtained.

[0064] S3. Intermediate code generation, the specific steps are as follows:

[0065] Based on the official documentation regarding the classes, properties, and methods of the framework before and after migration, a one-to-one mapping is established. Based on the corresponding properties and methods of the basic controls in the control library, migration rules and corresponding processing logic code are generated.

[0066] The code rewriter, based on the generated abstract code tree and semantic analysis data, modifies the type names, property names, and method names that need modification in the base class by tracing type definitions and resolving references. Simultaneously, it iterates through all locations where the class is called, modifying each one individually. The code rewriter determines the locations requiring modification based on the abstract syntax tree, performing a step-by-step matching at the tree node, statement, and code block levels, making modifications and cascading changes, and generating intermediate code after processing.

[0067] Step S3 mainly consists of extracting the content of the framework control library before and after migration, generating migration rules, refactoring code, and handling related relationships.

[0068] During the framework control library content extraction phase, a web crawler framework was used to crawl official documentation, open-source projects, and community Q&A information from both the source framework software library and the target migration software library. The crawled information was then processed. Simultaneously, to ensure that the migration rules corresponded one-to-one at the declaration space, control level, class level, method level, and statement body level, the methods and properties of each front-end control in the original framework were implemented to correspond with the methods and properties of the target migration framework.

[0069] During the migration rule generation phase, by comparing the crawled content with the control library code of the original framework and the target framework before and after migration, a one-by-one matching and mapping is performed based on coarse-grained levels such as project file level, control level, declaration space level, class level, and method body level, and a fine-grained mapping is performed based on statement body level and identifier level to generate migration rules based on statement level.

[0070] like Figure 2As shown, specifically, three types of migration rules are summarized: (1) Single-point migration. This type of rule includes the migration of a single syntax node in the AST, such as modifying variable attribute names, function names, etc. This type uses a code rewriter to traverse the syntax tree for rewriting. (2) Single-statement migration: This type of migration involves the migration of a single statement. A code rewriter is used to traverse the code statement by statement and use regular expressions for matching and migration. (3) Complex structure migration. This type of migration is often highly variable and complex, and special processing is performed for each migration. It is achieved by directly modifying the AST.

[0071] During the code refactoring phase, the semantic model generated by the original front-end framework project is loaded, and the statements to be migrated are located based on the AST nodes. The location process uses regular expression matching, code analysis tools to match nodes one by one, and other methods to generate the located statements into the target framework statements.

[0072] During the association processing phase, the declarations and calls of related variables, functions, and classes in the context are recursively searched based on the symbol table. For example, based on the declaration method of a class, the inheritance relationship of the class is searched to locate the first declaration of the method and modify the relevant content.

[0073] S4. Model customization, the specific steps are as follows:

[0074] Because LLM lacks domain-specific training data, and pre-trained data is time-sensitive while the content in that domain is constantly being updated, the content generated by the front-end framework in the migration code for the vertical domain is often very inaccurate. Therefore, it is necessary to encapsulate the process of calling LLM, which can be divided into: connecting to external databases, model fine-tuning, incremental training to optimize the model, input preprocessing, and chained calls.

[0075] In connecting to external databases, the llama-index library is called. Through the SimpleDirectoryReader data loader, the official documentation information files of the entire framework are loaded and updated continuously. All documents are then indexed by GPTSimpleVectorIndex. All documents are segmented and converted into vectors and stored as an index. External data is imported by calling the index to generate corresponding answers.

[0076] Due to token limitations, complete external data cannot be directly imported. Instead, the external data needs to be segmented using SpacyTextSplitter to generate segmented summaries. These summaries are then further subdivided to create a tree-like index. Each node in the tree represents a summary of its subtree content, and the root node represents the summary of the entire document.

[0077] For official documents, which often contain tables and images, a multimodal model is introduced to recognize the images. The ImageParser class is introduced, which generates text based on an OCR scanning model. By specifying a FileExtractor, the corresponding image is parsed into text by ImageParser, and finally converted into a vector for retrieval.

[0078] In model fine-tuning, information is summarized based on external data, relevant documents, and internal manual transfer rules. The model is trained on the basis of the original model, and the original parameters are changed to obtain a fine-tuned new model.

[0079] S5. Code file processing, the specific steps are as follows:

[0080] By splitting the project file content into statement bodies and generating the parameters and functional information of the code, a summary of the split code is generated. The summaries of each split code segment are then summarized to construct a tree index. Each tree node is a summary of the subtree code segment, and the final root node is a summary of the project file's functions and parameters.

[0081] S6. Chained calls, the specific steps are as follows:

[0082] In chained calls, LLM provides two core interfaces: Completion and Embedding. By including historical request records in the Prompt, LLM can correctly generate content based on the context. By pre-saving the index of Embedding, and through multiple rounds of dialogue, the answer returned in the previous round is used as new input to gradually determine the function of each project file.

[0083] S7. Final code generation, the specific steps are as follows:

[0084] The intermediate code is based on the content of the original framework code, and the target framework code is based on the functionality of the original framework code. Optimizing the intermediate code with the target framework code can optimize the parts that the code rewriter could not handle that do not conform to the syntax while maintaining the structure of the original code, and generate error logs and optimization suggestions for subsequent developers to handle.

[0085] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hybrid front-end framework migration method based on AST and LLM, characterized in that, Includes the following steps: (1) Load the original front-end framework project, identify the project files, and divide the functional project code into logic processing code and user interface definition code; (2) Lexical analysis, syntactic analysis and semantic extraction, the specific steps are as follows: (2-1) Lexical analysis, which decomposes the logic processing code and user interface definition code into a series of lexical units; (2-2) Syntax analysis: Based on lexical analysis, syntax analysis is performed to organize lexical units into an abstract syntax tree (AST) to represent the hierarchical structure of the code and the nesting relationships between code elements. (2-3) Semantic analysis: Semantic analysis is performed on the basis of AST, including type checking, scope analysis, symbol resolution, inheritance and implementation analysis; (3) Code rewriter processing, the specific steps are as follows: (3-1) Based on the control library of the original front-end framework and the target framework, write the implementation code of each method and property based on the control library; (3-2) Based on the correspondence between the properties and method calls of each control before and after migration, the migration rules are summarized according to the complexity. The migration rules are divided into: single point migration, single statement migration, and complex structure migration. (3-3) Implement a code rewriter based on the semantic analysis AST model and transfer rules to generate intermediate code; (4) Incremental training and optimization to generate a customized large model LLM, using the customized large model LLM to obtain the code function description of the original project file, and further generate the target framework code; (5) Based on the intermediate code, generate syntax error reports and processing suggestions through a customized large model LLM; optimize the intermediate code using the target framework code to generate the final migration code.

2. The hybrid front-end framework migration method based on AST and LLM according to claim 1, characterized in that, The specific process of step (1) is as follows: The code compilation analysis tool loads the pre-compiled version of the entire project, identifies the contents of the project files, and categorizes the files into logic processing files, user interface definition files, resource files, configuration files, and other files. It then extracts the logic processing code files and user interface definition code files and generates a mapping relationship between them.

3. The hybrid front-end framework migration method based on AST and LLM according to claim 1, characterized in that, In step (2-1), the logic processing code is decomposed into keywords, identifiers, and operators, and the user interface definition code is decomposed into label names, attribute names, and attribute values.

4. The hybrid front-end framework migration method based on AST and LLM according to claim 1, characterized in that, In steps (2-3), during the type checking phase, the data types of variables, expressions, and values ​​in the code will be determined; during the scope analysis phase, the scope of variables, functions, and classes will be identified, and the declaration and use of variables within the scope will be checked; during the symbol resolution phase, a symbol table will be built to store information about variables, functions, and classes, and symbols will be identified and declarations and references will be found; during the inheritance and implementation analysis phase, the inheritance relationships and interfaces between classes will be analyzed.

5. The hybrid front-end framework migration method based on AST and LLM according to claim 1, characterized in that, In step (3-2), for single-point migration, the migration is updated by traversing the syntax tree; for single-statement migration, the migration is achieved by matching the target statement with regular expressions and extracting the call information; for complex structure migration, the migration is achieved by modifying the syntax tree for each rule.

6. The hybrid front-end framework migration method based on AST and LLM according to claim 1, characterized in that, In step (3-3), the code rewriter is generated based on the AST operation tree nodes to modify, move, and reorganize the code; and it is divided into regular rule rewriter, special rule rewriter, and subsequent optimization rewriter based on migration rules.

7. The hybrid front-end framework migration method based on AST and LLM according to claim 1, characterized in that, The specific process of step (4) is as follows: (4-1) Load external data, relevant documents and migration rule summary information as data sources into the form of large model reading, divide the data into specified sizes and store them in the vector database in the form of embedding, pass them to the large model, and generate a customized large model LLM. (4-2) Statistically analyze all code to be migrated in the project, determine whether the token limit has been exceeded, and split the original project files based on semantic analysis information; (4-3) Based on the correspondence between the logic processing code and the user interface definition code, as well as the context call relationship, merge and upload the customized large model LLM to obtain the code function description of the original project file; (4-4) Based on the functional description, call the customized large model LLM to generate the target framework code.

8. The hybrid front-end framework migration method based on AST and LLM according to claim 7, characterized in that, In step (4-1), the data source is divided into document blocks of a specified size and stored in the vector database in an embedded form. When inputting a prompt, the segmented document is retrieved from the vector database. By comparing the cosine similarity, the document block in the vector database that is similar to the prompt question is found. The document block is passed to the LLM model, and the prompt containing the question and the document block is used to generate the answer.

9. The hybrid front-end framework migration method based on AST and LLM according to claim 1, characterized in that, In step (5), the intermediate code is based on the content of the original framework code, and the target framework code is based on the functionality of the original framework code. The intermediate code is optimized using the target framework code. While maintaining the structure of the original code, the code rewriter optimizes the parts that do not conform to the syntax that it cannot handle, and generates error logs and optimization suggestions for subsequent developers to handle.

Images