Intelligent programming auxiliary method and system based on multi-mode AI language model

Abstract

The invention discloses an intelligent programming auxiliary method and system based on a multi-modal AI language model, and belongs to the technical field of programming auxiliary tools. The method comprises the following steps: a multi-modal input processing stage; a dynamic context modeling stage; a hierarchical semantic analysis stage; in the code generation stage, codes are generated in two stages by adopting a Codex-Plus large model; the reinforcement learning driven code optimization stage is used for carrying out multi-objective optimization and reward function design on the codes generated in the code generation stage; a multi-dimensional feedback stage; an interaction and visualization stage; code semantic deep analysis, dynamic context sensing, multi-target optimization generation and real-time interactive feedback are realized by fusing code texts, natural language description, developer behavior data and a domain knowledge graph, and programming efficiency and code quality can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of programming assistance tools, and in particular to an intelligent programming assistance method and system based on a multimodal AI language model. Background Art

[0002] With the continuous increase in the complexity and scale of software development, developers face higher requirements for efficiency and quality. Traditional programming assistance tools, such as code completion, static analysis tools, and integrated development environments (IDEs), although they play a certain role in improving development efficiency, still have significant deficiencies in semantic understanding, context awareness, and multi-objective optimization. These limitations are mainly reflected in the following aspects:

[0003] Shallow semantic understanding. Existing tools mainly rely on syntax rules and pattern matching (such as regular expressions) to provide code completion or error detection functions. However, these tools cannot understand the deep business logic of the code, such as the relevance between function intent and requirement documents. When a developer writes a "user login" function, traditional tools can only provide syntactically correct code snippets, but cannot ensure that the code logic is consistent with the business requirements.

[0004] Fragmented context awareness. Traditional tools usually provide suggestions only based on local code snippets, lacking the association with the global project structure, developer historical habits, and external knowledge bases. For example, when refactoring code, the tool cannot recognize the dependency relationships of other modules in the project, which may lead to new problems in the refactored code.

[0005] Single optimization goal. Existing automated tools generally optimize for a single metric (such as performance or security), lacking the ability to dynamically balance between multiple objectives (performance, security, readability). For example, some tools may focus on improving code performance but ignore code maintainability or security, resulting in potential risks in actual applications.

[0006] In recent years, the rapid development of artificial intelligence technology has brought new possibilities to programming assistance tools. For example, code generation models based on deep learning (such as GitHub Copilot) can generate code snippets according to natural language descriptions, but still have deficiencies in semantic understanding depth, context relevance, and multi-objective optimization. In addition, existing AI tools are prone to generate results with correct syntax but logical errors when generating code, and lack the ability to adapt to developers' personalized needs.

[0007] In view of the above problems, the present invention proposes an intelligent programming assistance method and system based on a multi-modal AI language model. By integrating code text, natural language descriptions, developer behavior data, and domain knowledge graphs, it realizes in-depth code semantic analysis, dynamic context awareness, multi-objective optimization generation, and real-time interactive feedback, which can significantly improve programming efficiency and code quality. Summary of the Invention

[0008] The purpose of this part is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this part, as well as in the abstract and title of the present invention, to avoid obscuring the purpose of this part, the abstract, and the title. However, such simplifications or omissions shall not be used to limit the scope of the present invention.

[0009] Therefore, the object of the present invention is to provide an intelligent programming assistance method and system based on a multi-modal AI language model. By integrating multi-modal data such as code text, natural language descriptions, developer behavior data, and domain knowledge graphs, it realizes in-depth code semantic analysis, dynamic context awareness, multi-objective optimization generation, and real-time interactive feedback, significantly improving programming efficiency and code quality. Moreover, through the multi-modal input data fusion mechanism, hierarchical semantic analysis framework, reinforcement learning-driven multi-objective optimization, and natural language interactive programming, it can effectively solve the limitations of traditional tools and provide developers with a more intelligent and efficient programming assistance experience.

[0010] To solve the above technical problems, the present invention provides an intelligent programming assistance method based on a multi-modal AI language model, adopting the following technical solutions: including:

[0011] Multi-modal input processing stage: Collect multi-modal data of open-source code files, natural language annotations / requirement documents, developer operation logs, and domain knowledge graphs, and use syntax parsing tools, the Bidirectional Encoder Representations from Transformers (BERT), long short-term memory networks, and graph database tools to process the multi-modal data respectively;

[0012] Dynamic context modeling stage: Integrate the code dependency graph, text semantics, and domain knowledge-related results, construct a unified semantic network in the graph database, and use graph neural networks to achieve cross-modal joint reasoning, forming an intelligent context environment that supports real-time incremental updates;

[0013] Hierarchical semantic analysis stage: Through progressive hierarchical analysis, deeply understand the user's intention from syntax to business logic. When mapping business logic, calculate the semantic similarity between code segments and requirement documents through siamese neural networks to align technical implementation with business requirements;

[0014] Code generation stage: Use the Codex-Plus large model to generate code in two stages;

[0015] Reinforcement learning-driven code optimization phase: used to perform multi-objective optimization and reward function design on the code generated in the code generation phase;

[0016] Multi-dimensional feedback phase: simultaneously perform three types of analysis tasks, including analyzing and performing static code checks, dynamic performance profiling, and security vulnerability scanning, to form a multi-dimensional quality assessment report;

[0017] Interaction and visualization phase: Developers, according to the content of the evaluation report, through natural language interaction, after the developer inputs instructions, parse the operation type and parameters through the Bidirectional Encoder Representations from Transformers model BERT + Conditional Random Field CRF model, convert them into specific code modifications, and finally present suggestions hierarchically on the IDE panel.

[0018] Optionally,

[0019] The open-source code file processing method: real-time capture code snippets written by developers through an IDE plugin, use a syntax parsing tool to generate an abstract syntax tree and a data flow diagram, and extract and store structured information such as variable scopes and function call chains;

[0020] The natural language annotation / requirement document processing method: First, perform data preprocessing on the document, including word segmentation, adding special markers, and annotating entities in BIO format; then load the pre-trained Bidirectional Encoder Representations from Transformers model BERT and adjust the output layer according to task requirements, optimize the model through a fine-tuning strategy, and finally, use the bidirectional attention mechanism of the Bidirectional Encoder Representations from Transformers model BERT to capture the deep semantics of the text and output structured information to achieve entity recognition and relationship extraction;

[0021] The developer operation log processing method: First, preprocess these code operation logs to obtain parameters related to recording code editing history, debugging behavior, and suggestion acceptance rate, then build a long short-term memory network time series model, set appropriate number of layers and hidden unit quantity parameters, then use the preprocessed dataset to train the model, and optimize the model performance by continuously adjusting parameters; after training, evaluate the model using the test set; finally, according to the output results of the trained model, analyze the coding style preferences of different developers;

[0022] The domain knowledge graph processing method: construct a knowledge graph containing API documents, security rules, and design patterns, and use a graph database to store entity relationships to support dynamic retrieval and association.

[0023] Optionally, the dynamic context modeling phase specifically includes:

[0024] First, integrate multi-modal multi-source heterogeneous data. Use a syntax parsing tool to extract the abstract syntax tree and data flow graph from code files, and construct a project-level code dependency graph. Parse natural language annotations and requirement documents through a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model, extract functional entities, and establish mappings with code class / method nodes. Analyze developer operation logs using a Long Short-Term Memory (LSTM) network model, quantify coding style preferences, and associate them with specific code entities.

[0025] At the same time, dynamically link the pre-constructed domain knowledge graph to code entities through graph embedding technology. Then, use graph embedding technology to dynamically link the entity relationships in the pre-constructed domain knowledge graph to the code dependency graph, forming a semantic network that integrates multi-modal inputs.

[0026] Finally, based on the attention mechanism and graph neural network, maintain a scalable context representation space in real time to support the collaborative reasoning of four-dimensional data: code entities, functional requirements, developer behavior, and domain knowledge, and finally output a global context model with dynamic perception capabilities.

[0027] Optionally, the hierarchical semantic parsing stage is divided into three progressive analyses, including: First, perform syntax-level parsing. Use the Tree-sitter tool to generate the abstract syntax tree (AST) of the code, extract basic syntax units from the AST, and mark the types of code blocks. Then, perform data flow analysis. Based on the AST, construct a variable propagation graph, trace the variable life cycle and data dependency relationships, identify potential problems, and generate a data flow anomaly report. Finally, perform business logic mapping. Use a Siamese neural network to construct a two-tower structure to achieve heterogeneous semantic alignment. First, input the requirement text into the text encoding tower of the BERT model, and extract the marked deep semantic vector to represent the business intention. At the same time, encode the code segment into a vector through a graph neural network, integrating the control flow features extracted from its AST and the data flow dependency relationships. The two-tower network aligns the semantic spaces through a shared weight mechanism, calculates the matching degree between the code and requirement vectors using a cosine similarity layer, and optimizes the model parameters based on the corresponding loss function. After training, combine the domain knowledge graph to perform logical verification on the high-similarity matching results, and finally generate a mapping relationship matrix between the code segment and the requirement entity.

[0028] Optionally, the code generation stage specifically includes:

[0029] Based on the Codex-Plus large model, a multi-modal input encoding layer is introduced into the large model to convert the Token sequence into a vector representation, preserving the syntactic structure information. At the same time, a graph attention network is used to embed the Abstract Syntax Tree (AST) nodes to capture the code-level relationships, and a Bidirectional Encoder Representations from Transformers (BERT) is used to extract the functional intent vectors in the requirement description. A cross-attention decoding mechanism is used to dynamically fuse syntactic constraints and business semantics during the decoding phase.

[0030] The two-stage code generation method includes first generating the output function framework, determining the input and output interfaces, and then filling in the details and implementing based on the reinforcement learning strategy to generate the complete code logic.

[0031] Optionally, the multi-objective optimization and reward function design includes:

[0032] Performance reward: Based on static time complexity analysis and dynamic execution time measurement, the calculation formula is as follows:

[0033]

[0034] Among them, the actual execution time is obtained by measuring the code during the code implantation time, and the execution time of the benchmark algorithm is obtained through theoretical calculation.

[0035] Security reward: Detect the number of vulnerabilities through a static scanning tool, and the calculation formula is as follows:

[0036]

[0037] Among them, the number of vulnerabilities is obtained through a static scanning tool, and at the same time, the number of vulnerabilities that match the vulnerability standard pattern.

[0038] Readability reward: Based on code complexity metrics (such as cyclomatic complexity, Halstead metrics), the calculation formula is as follows:

[0039]

[0040] Among them, the cyclomatic complexity measures the complexity of the code branch logic.

[0041] The final reward function is the weighted sum of the three:

[0042] R = αR perf + βR sec + γR read

[0043] α + β + γ = 1

[0044] α, β, γ are weight coefficients and are dynamically adjusted according to project requirements.

[0045] Optionally, train the code generation model on the GitHub open-source code repository to learn common coding patterns; then perform online fine-tuning: during the developer's use, collect feedback signals in real time, and use the proximal policy optimization algorithm to update the model parameters to preferentially meet the constraints of the current project; in addition, regularly perform incremental training on the code generation to incorporate new domain knowledge and developer behavior data.

[0046] Optionally, execute three types of analysis tasks in parallel through a multi-threaded architecture and implement a feedback loop: First, the static analysis module calls the SonarQube rule engine to scan the code structure based on predefined code quality rules, locate whether the code is abnormal, and generate a refactoring suggestion map; at the same time, the dynamic performance profiling module deploys the code in an isolated Docker container and injects a load test script, monitors the runtime metrics through Prometheus, captures the peak CPU occupancy rate at the function level and memory leak points, and combines the flame graph to locate inefficient algorithms; in parallel, the security scanning module loads the Semgrep rule library, detects risky code through abstract syntax tree pattern matching, and associates the predefined repair strategies in the knowledge graph; after the three types of analysis results are normalized, a comprehensive quality report is generated.

[0047] Optionally, the IDE panel is displayed according to priority, where P0 is urgent: security vulnerability repair; P1 is high-priority: performance optimization; P2 is a suggestion: code style adjustment.

[0048] An intelligent programming assistance system based on a multi-modal AI language model, including:

[0049] A multi-modal input processing module, which is used to collect multi-modal data such as open-source code files, natural language annotations / requirement documents, developer operation logs, and domain knowledge graphs, and use syntax parsing tools, the bidirectional encoder representation model BERT based on transformers, long short-term memory networks, and graph database tools to process the multi-modal data respectively;

[0050] A dynamic context modeling module: Integrate the code dependency graph, text semantics, and domain knowledge-related results, build a unified semantic network in the graph database, and use graph neural networks to achieve cross-modal joint reasoning to form an intelligent context environment that supports real-time incremental updates;

[0051] A hierarchical semantic parsing module: Through progressive hierarchical analysis, deeply understand the user's intention from syntax to business logic. When mapping business logic, calculate the semantic similarity between code segments and requirement documents through a siamese neural network to align technical implementation with business requirements;

[0052] A code generation module: Use the Codex-Plus large model to generate code in two stages;

[0053] Reinforcement Learning Driven Code Optimization Module: Used for multi-objective optimization and reward function design of the code generated during the code generation stage;

[0054] Multi-Dimensional Feedback Module: Simultaneously performs three types of analysis tasks, including analyzing and performing static code checks, dynamic performance profiling, and security vulnerability scanning, to form a multi-dimensional quality assessment report;

[0055] Interaction and Visualization Module: Developers use natural language interaction according to the content of the assessment report. After the developer inputs an instruction, the operation type and parameters are parsed through the Bidirectional Encoder Representations from Transformers (BERT) + Conditional Random Field (CRF) model, converted into specific code modifications, and finally the suggestions are presented hierarchically on the IDE panel.

[0056] In summary, the present invention includes at least one of the following beneficial effects:

[0057] By integrating multi-modal data such as code text, natural language descriptions, developer behavior data, and domain knowledge graphs, it realizes in-depth code semantic parsing, dynamic context awareness, multi-objective optimization generation, and real-time interactive feedback, significantly improving programming efficiency and code quality. And through the multi-modal input data fusion mechanism, hierarchical semantic parsing framework, reinforcement learning driven multi-objective optimization, and natural language interactive programming, it can effectively solve the limitations of traditional tools and provide a more intelligent and efficient programming assistance experience for developers. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0059] Figure 1 It is a schematic flowchart of the intelligent programming assistance method based on the multi-modal AI language model of the present invention;

[0060] Figure 2 It is a schematic framework diagram of the intelligent programming assistance system based on the multi-modal AI language model of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0062] In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper", "lower", "inner", "outer", "top / bottom end", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance.

[0063] In the description of the present invention, it should be noted that unless otherwise clearly specified and limited, the terms "installation", "provided with", "sheathed / connected", "connection", etc. should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0064] Embodiment 1

[0065] Referring to Figure 1 , the present invention discloses an intelligent programming assistance method based on a multi-modal AI language model, including:

[0066] Multi-modal input processing stage: By collecting multi-modal data such as existing open-source code files, natural language annotations / requirement documents, developer operation logs, and domain knowledge graphs, and respectively using grammar parsing tools, the bidirectional encoder representation model BERT based on transformers, long short-term memory networks, and graph databases and other tools to intelligently process the multi-modal data;

[0067] Dynamic context modeling stage: Integrate relevant results such as code dependency graphs, text semantics, and domain knowledge, construct a unified semantic network in a graph database, and use graph neural networks to achieve cross-modal joint reasoning to form an intelligent context environment that supports real-time incremental updates;

[0068] Furthermore, in the multi-modal input processing stage, the collection and preprocessing of four types of input data are mainly carried out, as shown in the following table:

[0069]

[0070] Among them, for the collected code files, the code snippets written by developers are captured in real time through an IDE plugin, and an abstract syntax tree and a data flow graph are generated using a grammar parsing tool (such as Tree-sitter), and structured information such as variable scopes and function call chains are extracted and stored.

[0071] For the collected natural language annotations and requirement documents, first, perform data preprocessing on the documents, including word segmentation, adding special tokens, and annotating entities in the BIO format; then load the pre-trained BERT model and adjust the output layer according to the task requirements, and optimize the model through the fine-tuning strategy. Finally, utilize the bidirectional attention mechanism of the Bidirectional Encoder Representations from Transformers (BERT) model to capture the deep semantics of the text and output structured information (such as entities, relation triples, or key phrases), thereby realizing entity recognition and relation extraction. For example, extract the functional entity "user login" from the annotation "Implement the user login function" and associate it with the AuthService class in the code.

[0072] For the collected developer operation logs, first preprocess these code operation logs to obtain relevant parameters such as recording the code editing history (such as the function renaming frequency), debugging behavior (such as the breakpoint setting location), and suggestion acceptance rate. Then construct a long short-term memory network time series model, set appropriate parameters such as the number of layers and the number of hidden units, and then use the preprocessed dataset to train the model and optimize the model performance by continuously adjusting the parameters. After training, evaluate the accuracy and reliability of the model using the test set. Finally, based on the output results of the trained model, analyze the coding style preferences of different developers (such as the tendency to use recursive algorithms), etc.

[0073] For the collected domain knowledge graphs, construct a knowledge graph containing API documents, security rules, and design patterns, and use a graph database to store entity relationships to support dynamic retrieval and association.

[0074] Then construct a dynamic context modeling engine based on the input multi-type data to achieve global semantic association. First, integrate multi-modal multi-source heterogeneous data, use a syntax parsing tool to extract the abstract syntax tree and data flow graph from the code file, and construct a project-level code dependency graph; parse the natural language annotations and requirement documents through a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model, extract functional entities, and establish mappings with code class / method nodes; use a long short-term memory network model to analyze developer operation logs, quantify coding style preferences, and associate them with specific code entities; at the same time, dynamically link the pre-constructed domain knowledge graph (such as security rules, API documents) to the code entities through graph embedding technology; then dynamically link the entity relationships in the pre-constructed domain knowledge graph to the code dependency graph through graph embedding technology to form a semantic network integrating multi-modal inputs. Finally, based on the attention mechanism and graph neural network, the dynamic context modeling engine maintains a scalable context representation space in real time, supports the collaborative reasoning of four-dimensional data of code entities, functional requirements, developer behavior, and domain knowledge, and finally outputs a global context model with dynamic perception ability to provide unified and linked semantic support for scenarios such as code intelligent recommendation and defect prediction.

[0075] Hierarchical semantic parsing stage: First, through progressive hierarchical analysis, deeply understand the user's intention from grammar to business logic. When mapping business logic, calculate the semantic similarity between code segments and requirement documents through a twin neural network to align technical implementation with business requirements, laying a good foundation for code generation;

[0076] Code generation stage: When generating code, preferably select the Codex-Plus large model, introduce a multi-modal input encoding layer, integrate code text encoding, abstract syntax tree graph embedding, and semantic understanding of natural language, use the cross-attention mechanism to fuse grammar rules and business semantics, and then generate code in two stages. First, generate the function framework to define the interface, and then fill in the detailed logic through a reinforcement learning strategy to achieve double guarantees of formal correctness and functional completeness;

[0077] Furthermore, semantic parsing is divided into three-layer progressive analysis to ensure a deep understanding of the user's intention from grammar to business logic. First, perform grammar-level parsing. You can use currently popular tools (such as: Tree-sitter) to generate the abstract syntax tree (AST) of the code, extract basic grammar units (such as loop structures, conditional branches) from the AST, and mark the types of code blocks (such as function definitions, class declarations); then perform data flow analysis. Based on the AST, construct a variable propagation graph, track the variable life cycle and data dependency relationships, identify potential problems (such as uninitialized variables, type conflicts), and generate a data flow exception report; finally, perform business logic mapping. Use a twin neural network to construct a two-tower structure to achieve heterogeneous semantic alignment. First, input the requirement text into the text encoding tower of the bidirectional encoder representation model BERT based on the transformer to extract the deep semantic vector marked by [CLS] to represent the business intention; at the same time, encode the code segment into a vector through a graph neural network, integrating the control flow features and data flow dependency relationships extracted from its abstract syntax tree; the two-tower network aligns the semantic space through a shared weight mechanism, calculates the matching degree between the code and the requirement vector using a cosine similarity layer, and optimizes the model parameters based on the corresponding loss function; after training, combine the domain knowledge graph to perform logical verification on the high-similarity matching results, and finally generate a mapping relationship matrix between the code segment and the requirement entity to achieve an interpretable association from technical implementation to business goals.

[0078] Furthermore, the multi-modal code generation model is preferably based on the Codex-Plus large model. Codex is an AI code generation training model under OpenAI. The Codex-Plus large model is an extended architecture based on GPT-4. To better meet the requirements of multi-modal input information, a multi-modal input encoding layer is introduced into the framework to convert the Token sequence (a Token sequence refers to a sequence composed of a series of basic units obtained by splitting the code text according to specific rules. In the code generation process, the code is regarded as a string, and then it is split into individual Tokens using a tokenizer. These Tokens can be words, punctuation marks, code keywords, etc.) into a vector representation, preserving the syntactic structure information. At the same time, a graph attention network is used to embed AST nodes to capture the code-level relationships, and a functional intent vector in the requirement description is extracted through the Bidirectional Encoder Representations from Transformers (BERT); a cross-attention decoding mechanism is used in the code model to dynamically fuse syntactic constraints (such as indentation rules) and business semantics (such as "transaction rollback needs to be supported") during the decoding phase to ensure that the generated code meets both formal correctness and functional matching; finally, a two-stage generation method is adopted. First, an output function framework (such as def process_order(order:Order)->bool:) is generated to determine the input and output interfaces, and then details are filled in. Based on the reinforcement learning strategy, specific implementations (such as using ThreadPoolExecutor instead of manual thread management) are selected to generate the complete code logic.

[0079] Reinforcement learning-driven code optimization stage: To address the problem that traditional code optimization tools usually optimize for a single dimension (such as performance or security), the intelligent assistance method provided by the present invention takes into account the needs in actual software development and comprehensively weighs multiple key attributes. When optimizing the code generated in the second stage, multi-objective optimization and reward function design are preferably adopted, and the optimization objectives cover performance, security, and readability; in addition, to further improve the applicability of the method provided by the present invention, the code generation model also needs to be fine-tuned online and continuously learned to continuously improve the accuracy of the method provided by the present invention.

[0080] The adoption of multi-objective optimization and reward function design includes:

[0081] Performance reward: Based on static time complexity analysis (Big O notation) and dynamic execution time measurement, the calculation formula is as follows:

[0082]

[0083] Among them, the actual execution time can be obtained by measuring the code at the code implantation time, and the execution time of the benchmark algorithm can be obtained through theoretical calculation.

[0084] Security Reward: The number of vulnerabilities detected by a static scanning tool (such as vulnerabilities with CWE numbers) is calculated as follows:

[0085]

[0086] Among them, the number of vulnerabilities is obtained by a static scanning tool, and at the same time, the number of vulnerabilities that match the CWE vulnerability standard pattern.

[0087] Readability Reward: Based on code complexity metrics (such as cyclomatic complexity, Halstead metrics), the calculation formula is as follows:

[0088]

[0089] Among them, cyclomatic complexity measures the complexity of the code branch logic. Code with high complexity is more error-prone and difficult to maintain, and can be obtained by implanting existing tool functions.

[0090] The final reward function is the weighted sum of the three as follows:

[0091] R = αR perf + βR sec + γR read

[0092] α + β + γ = 1

[0093] α, β, γ are weight coefficients, which can be dynamically adjusted according to project requirements (for example, β can be set to 0.5 in security-sensitive scenarios).

[0094] Furthermore, in order to improve the accuracy and reliability of the model, the model needs to be trained and fine-tuned. During offline pre-training, the basic generation model can be trained on the GitHub open-source code library (filtering License restrictions) to learn common coding patterns; then online fine-tuning: during the developer's use, real-time feedback signals are collected (such as acceptance / rejection suggestions, manual modification records), and the proximal policy optimization (PPO) algorithm is used to update the model parameters, giving priority to meeting the current project constraints (such as disabling specific insecure APIs); in addition, the model also needs to be incrementally trained regularly, incorporating new domain knowledge (such as updated API documentation) and developer behavior data to achieve the purpose of continuously improving the accuracy and reliability of the code generation model.

[0095] Multi-dimensional Feedback Stage: By performing real-time analysis on the output results of the method provided by the present invention and conducting static code inspection, dynamic performance profiling, and security vulnerability scanning, a multi-dimensional quality assessment report is formed;

[0096] Interaction and Visualization Phase: Developers can conduct natural language interaction with the system provided by the present invention based on the evaluation report. After the developers input instructions, the system parses the operation type and parameters through the Bidirectional Encoder Representations from Transformers (BERT) + Conditional Random Field (CRF) model, and then converts them into specific code modifications. Finally, suggestions are presented hierarchically on the IDE panel, realizing a closed-loop interaction from problem detection to intelligent repair. Among them, the Conditional Random Field (CRF) model enforces the legality of the output sequence by modeling the label transition probability, complementing the context semantic representation of the Bidirectional Encoder Representations from Transformers (BERT) model, and can effectively solve problems such as illegal label transfer, long-distance dependence, and semantic consistency, ensuring that the generated code modification instructions are logically associated with the parameters. This design significantly improves the accuracy and reliability of natural language instruction parsing.

[0097] Furthermore, the method provided by the present invention simultaneously executes three types of analysis tasks to form a comprehensive quality assessment.

[0098] During static analysis, it is preferred to use the SonarQube rule engine to detect code anomalies (such as duplicate code, overly long functions) and give refactoring suggestions (such as extracting methods, merging conditional expressions); during dynamic performance profiling, it is preferred to execute the generated code in a Docker sandbox environment, collect the CPU / memory occupancy rate and the heat map of function execution time, and identify performance bottlenecks (such as inefficient loops with high-frequency calls); during security scanning, it is preferred to call Semgrep for pattern matching to detect security vulnerabilities (such as hard-coded keys, SQL injection risks), and associate the repair solutions in the knowledge graph (such as recommending the use of parameterized queries).

[0099] Furthermore, the method provided by the present invention executes three types of analysis tasks in parallel through a multi-threaded architecture and realizes a feedback closed-loop: First, the static analysis module calls the SonarQube rule engine, scans the code structure based on predefined code quality rules (such as CWE, ISO5055 standard), locates whether the code is abnormal (such as duplicate code blocks, functions exceeding 50 lines), and generates a refactoring suggestion map (such as splitting nested loops into independent methods, or replacing conditional branches with the strategy pattern); at the same time, the dynamic performance profiling module deploys the code in an isolated Docker container and injects load test scripts, monitors runtime metrics through Prometheus, captures the peak CPU occupancy rate at the function level (such as a single function occupying more than 30%), and memory leak points (such as an unreleased database connection pool), and combines the flame graph to locate inefficient algorithms (such as O(n2) sorting); in parallel, the security scanning module loads the Semgrep rule library (including OWASP Top 10 vulnerability patterns), detects risky code (such as API keys stored in plain text, unfiltered user input) through abstract syntax tree pattern matching, and associates the predefined repair strategies in the knowledge graph (such as replacing string concatenation queries with PreparedStatement). After the three types of analysis results are normalized, a comprehensive quality report (including defect priority labels) is generated.

[0100] Furthermore, the method provided by the present invention can perform natural language interactive programming and instruction parsing. The developer inputs a natural language instruction (such as "asynchronize this function and limit the maximum number of retries to 3 times"), and uses a Bidirectional Encoder Representations from Transformers (BERT) + Conditional Random Field (CRF) model for intent classification (operation type: asynchronous transformation; parameter: number of retries = 3); for code modification execution, the instruction is converted into specific code operations (such as replacing the synchronous function fetch_data() with async_fetch_data() and adding a retry decorator @retry(max_attempts = 3)); the IDE panel displays according to priority, P0 (urgent): security vulnerability repair (such as unencrypted sensitive data transmission); P1 (high priority): performance optimization (such as replacing an O(n2) algorithm with an O(n log n) implementation); P2 (recommended): code style adjustment (such as normalizing variable naming).

[0101] Embodiment 2

[0102] Referring to Figure 2 , based on the same concept as in the above Embodiment 1, it further includes an intelligent programming assistance system based on a multi-modal AI language model, including:

[0103] The multi-modal input processing module is used to collect multi-modal data of open-source code files, natural language annotations / requirement documents, developer operation logs, and domain knowledge graphs, and respectively use syntax parsing tools, the Bidirectional Encoder Representations from Transformers model BERT (Bidirectional Encoder Representations from Transformers), long short-term memory networks, and graph database tools to process the multi-modal data;

[0104] The dynamic context modeling module: integrates the results related to code dependency graphs, text semantics, and domain knowledge, constructs a unified semantic network in the graph database, and uses graph neural networks to achieve cross-modal joint reasoning to form an intelligent context environment that supports real-time incremental updates;

[0105] The hierarchical semantic parsing module: through progressive hierarchical analysis, deeply understands the user's intention from syntax to business logic. When mapping business logic, it calculates the semantic similarity between code segments and requirement documents through a siamese neural network to achieve the alignment of technical implementation and business requirements;

[0106] The code generation module: uses the Codex-Plus large model to generate code in two stages;

[0107] The reinforcement learning-driven code optimization module: is used to perform multi-objective optimization and reward function design on the code generated in the code generation stage;

[0108] The multi-dimensional feedback module: simultaneously performs three types of analysis tasks, including analyzing and performing static code checks, dynamic performance profiling, and security vulnerability scanning, to form a multi-dimensional quality assessment report;

[0109] The interaction and visualization module: developers use natural language interaction according to the content of the assessment report. After the developer inputs an instruction, it parses the operation type and parameters through the Bidirectional Encoder Representations from Transformers model BERT + Conditional Random Field CRF model, converts them into specific code modifications, and finally presents suggestions hierarchically on the IDE panel.

[0110] The present invention captures multimodal data in the development environment in real time through a multimodal input processing module, including code text, natural language comments, developer operation logs, and pre-built domain knowledge graphs; a dynamic context modeling module integrates the collected multimodal data to construct a global context model, covering code dependencies, developer habit portraits, and external knowledge associations; a hierarchical semantic parsing module performs in-depth semantic analysis on the code, progressing layer by layer from syntactic structures, data flow propagation to business logic mapping; a code generation module and a reinforcement learning-driven code optimization module generate candidate codes based on the parsing results and dynamically optimize among multiple objectives through reinforcement learning algorithms; a multi-dimensional feedback analysis module performs static checks, dynamic performance profiling, and security scans on the generated codes to form a comprehensive evaluation report; a multi-dimensional feedback module and an interaction and visualization module provide a natural language interaction interface and a visualization panel to support developers' real-time intervention and result viewing.

[0111] Each module transmits standardized data (such as context models and code semantic representations in JSON format) through a data bus, forming a complete link of "input → modeling → parsing → generation → optimization → feedback → interaction".

[0112] The above are all preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereby. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention shall be covered within the protection scope of the present invention.

Claims

1. An intelligent programming assistance method based on a multi-modal AI language model, characterized in that: Including: Multi-modal input processing stage: Collect multi-modal data such as open-source code files, natural language annotations / requirement documents, developer operation logs, and domain knowledge graphs, and use syntax parsing tools, the Transformer-based Bidirectional Encoder Representations from Transformers (BERT), long short-term memory networks, and graph database tools to process the multi-modal data respectively; Dynamic context modeling stage: Integrate the results related to code dependency graphs, text semantics, and domain knowledge, construct a unified semantic network in a graph database, and use graph neural networks to achieve cross-modal joint reasoning, forming an intelligent context environment that supports real-time incremental updates; Hierarchical semantic parsing stage: Through progressive hierarchical analysis, deeply understand the user's intention from syntax to business logic. When mapping business logic, calculate the semantic similarity between code segments and requirement documents through a Siamese neural network to align technical implementation with business requirements; Code generation stage: Use the Codex-Plus large model to generate code in two stages; Reinforcement learning-driven code optimization stage: Used to perform multi-objective optimization and reward function design on the code generated in the code generation stage; Multi-dimensional feedback stage: Simultaneously execute three types of analysis tasks, including analyzing and performing static code checks, dynamic performance profiling, and security vulnerability scanning, to form a multi-dimensional quality assessment report; Interaction and visualization stage: Developers use natural language interaction according to the content of the assessment report. After the developer inputs an instruction, the operation type and parameters are parsed through the Transformer-based Bidirectional Encoder Representations from Transformers (BERT) + Conditional Random Field (CRF) model, converted into specific code modifications, and finally the suggestions are presented hierarchically on the IDE panel.

2. The intelligent programming assistance method based on the multi-modal AI language model according to claim 1, wherein: The method for processing open-source code files: Real-time capture code segments written by developers through an IDE plugin, use a syntax parsing tool to generate an abstract syntax tree and a data flow graph, and extract and store structured information such as variable scopes and function call chains; The method for processing natural language annotations / requirement documents: First, perform data preprocessing on the document, including word segmentation, adding special markers, and annotating entities in BIO format; then load the pre-trained Transformer-based Bidirectional Encoder Representations from Transformers (BERT) and adjust the output layer according to task requirements, optimize the model through a fine-tuning strategy. Finally, use the bidirectional attention mechanism of the Transformer-based Bidirectional Encoder Representations from Transformers (BERT) to capture the deep semantics of the text, output structured information, and achieve entity recognition and relationship extraction; The method for processing developer operation logs: First, preprocess these code operation logs to obtain parameters related to recording code editing history, debugging behavior, and suggestion acceptance rate, then construct a long short-term memory network time series model, set appropriate parameters for the number of layers and the number of hidden units, and then use the preprocessed dataset to train the model, and optimize the model performance by continuously adjusting the parameters; after training, evaluate the model using the test set; finally, analyze the coding style preferences of different developers according to the output results of the trained model; The above-mentioned domain knowledge graph processing method: construct a knowledge graph including API documents, security rules, and design patterns, use a graph database to store entity relationships, and support dynamic retrieval and association.

3. The intelligent programming assistance method based on the multi-modal AI language model according to claim 2, wherein: The specific process of the dynamic context modeling stage includes: First, integrate multi-modal multi-source heterogeneous data. Use a syntax parsing tool to extract the abstract syntax tree and data flow graph from code files, and construct a project-level code dependency graph; parse natural language annotations and requirement documents through a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model to extract functional entities and establish mappings with code class / method nodes; use a Long Short-Term Memory (LSTM) network model to analyze developer operation logs, quantify coding style preferences, and associate them with specific code entities; At the same time, dynamically link the pre-constructed domain knowledge graph to code entities through graph embedding technology; then, through graph embedding technology, dynamically link the entity relationships in the pre-constructed domain knowledge graph to the code dependency graph to form a semantic network that integrates multi-modal inputs; Finally, based on the attention mechanism and graph neural network, continuously maintain an extensible context representation space in real time to support the collaborative reasoning of four-dimensional data: code entities, functional requirements, developer behavior, and domain knowledge, and finally output a global context model with dynamic perception capabilities.

4. The intelligent programming assistance method based on the multi-modal AI language model according to claim 3, characterized in that: The hierarchical semantic parsing stage is divided into three progressive analyses, including: First, perform syntax-level parsing. Use the Tree-sitter tool to generate the abstract syntax tree (AST) of the code, extract basic syntax units from the AST, and mark the types of code blocks; then perform data flow analysis. Based on the AST, construct a variable propagation graph to track the variable life cycle and data dependency relationships, identify potential problems, and generate a data flow anomaly report; finally, perform business logic mapping. Use a Siamese neural network to construct a two-tower structure to achieve heterogeneous semantic alignment. First, input the requirement text into the text encoding tower of the BERT model to extract the marked deep semantic vector to represent the business intention; at the same time, encode the code segment into a vector through a graph neural network, and fuse the control flow features extracted from its AST and the data flow dependency relationships; the two-tower network aligns the semantic space through a shared weight mechanism, calculates the matching degree between the code and requirement vectors using a cosine similarity layer, and optimizes the model parameters based on the corresponding loss function; after training, combine the domain knowledge graph to perform logical verification on the high similarity matching results, and finally generate a mapping relationship matrix between the code segment and the requirement entity.

5. The intelligent programming assistance method based on a multimodal AI language model according to claim 4, characterized in that: The specific process of the code generation stage includes: Based on the Codex-Plus large model, introduce a multi-modal input encoding layer into the large model to convert the Token sequence into a vector representation, retaining the syntax structure information. At the same time, use a graph attention network to embed the AST nodes to capture the code-level relationships, and extract the functional intention vector from the requirement description through the BERT model; use a cross-attention decoding mechanism to dynamically fuse syntax constraints and business semantics during the decoding stage. The two-stage code generation method includes first generating an output function framework, determining the input and output interfaces, and then filling in the details, selecting an implementation based on a reinforcement learning strategy, and generating a complete code logic.

6. The intelligent programming assistance method based on a multi-modal AI language model according to claim 1, characterized in that: The multi-objective optimization and reward function design mentioned above includes: Performance reward: Based on static time complexity analysis and dynamic execution time measurement, the calculation formula is as follows: Among them, the actual execution time is obtained by measuring the code at the code implantation time, and the execution time of the benchmark algorithm is obtained through theoretical calculation; Security reward: Detect the number of vulnerabilities through a static scanning tool, and the calculation formula is as follows: Among them, the number of vulnerabilities is obtained through a static scanning tool, and at the same time, the number of vulnerabilities that match the vulnerability standard pattern; Readability reward: Based on the code complexity metric, the calculation formula is as follows: Among them, the cyclomatic complexity measures the complexity of the code branch logic; The final reward function is the weighted sum of the three as follows: R = αR perf + βR sec + γR read α + β + γ = 1 α, β, and γ are weight coefficients, which are dynamically adjusted according to project requirements.

7. The intelligent programming assistance method based on the multimodal AI language model according to claim 6, characterized in that: Train the code generation model on the GitHub open-source code repository to learn common coding patterns; then perform online fine-tuning: during the use by developers, collect feedback signals in real time, and use the proximal policy optimization algorithm to update the model parameters to give priority to meeting the current project constraints; in addition, it is also necessary to incrementally train the code generation regularly, incorporating new domain knowledge and developer behavior data.

8. The intelligent programming assistance method based on a multi-modal AI language model according to claim 1, characterized in that: Execute three types of analysis tasks in parallel through a multi-threaded architecture and implement a feedback loop: First, the static analysis module calls the SonarQube rule engine to scan the code structure based on predefined code quality rules, locate whether the code is abnormal, and generate a refactoring suggestion graph; at the same time, the dynamic performance profiling module deploys the code in an isolated Docker container and injects a load test script, monitors the runtime metrics through Prometheus, captures the peak CPU occupancy rate at the function level and memory leak points, and combines the flame graph to locate inefficient algorithms; in parallel, the security scanning module loads the Semgrep rule library, detects risky code through abstract syntax tree pattern matching, and associates the predefined repair strategies in the knowledge graph; After the three types of analysis results are normalized, a comprehensive quality report is generated.

9. The intelligent programming assistance method based on a multi-modal AI language model according to claim 8, characterized in that: The IDE panel is displayed according to priority, where P0 is urgent: security vulnerability repair; P1 is high priority: performance optimization; P2 is a suggestion: code style adjustment.

10. An intelligent programming assistance system based on a multi-modal AI language model, characterized in that: Including: A multi-modal input processing module, which is used to collect multi-modal data such as open-source code files, natural language annotations / requirement documents, developer operation logs, and domain knowledge graphs, and respectively use a syntax parsing tool, a bidirectional encoder representation model BERT based on transformers, a long short-term memory network, and a graph database tool to process the multi-modal data; A dynamic context modeling module: Integrate the code dependency graph, text semantics, and relevant results of domain knowledge, build a unified semantic network in the graph database, and use graph neural networks to achieve cross-modal joint reasoning, forming an intelligent context environment that supports real-time incremental updates; Hierarchical Semantic Analysis Module: Through progressive hierarchical analysis, it deeply understands the user's intention from grammar to business logic. When mapping business logic, it calculates the semantic similarity between code segments and requirement documents through a twin neural network to achieve the alignment of technical implementation and business requirements; Code Generation Module: Uses the Codex-Plus large model to generate code in two stages; Reinforcement Learning Driven Code Optimization Module: Used to perform multi-objective optimization and reward function design on the code generated in the code generation stage; Multi-dimensional Feedback Module: Simultaneously performs three types of analysis tasks, including analyzing and performing static code checks, dynamic performance profiling, and security vulnerability scanning, to form a multi-dimensional quality assessment report; Interaction and Visualization Module: Developers interact in a natural language according to the content of the assessment report. After the developer inputs an instruction, it parses the operation type and parameters through a Bidirectional Encoder Representations from Transformers (BERT) + Conditional Random Field (CRF) model, converts them into specific code modifications, and finally presents the suggestions hierarchically on the IDE panel.

Images