AI code generation and defect repair method and system based on multi-modal input
Patent Information
- Application Number
- CN202610197832.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-11
- Publication Date
- 2026-10-30
AI Technical Summary
[0003]尽管现有技术已实现代码生成与缺陷修复的基础功能，但在实际工业应用中仍存在以下核心问题，难以满足企业级、领域化的研发需求：
[0052]1、需求解析更完整：支持文本、图像、表格多模态输入，通过跨模态融合实现需求信息的完整提取，提升了需求解析的准确性和全面性，大幅降低生成代码与需求的偏差。
Smart Images
Figure CN123011236A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and software development technology, specifically to a method and system for AI code generation and defect repair based on multimodal input. Background Technology
[0002] With the continuous expansion of software development scale and the rapid development of artificial intelligence technology, automatic code generation and defect repair technologies are playing an increasingly important role in software quality assurance. Currently, mainstream code generation and defect repair solutions are mainly divided into two categories: one is code generation models based on natural language input (such as CodeLlama, StarCoder, Copilot, etc.), which realize the transformation from text requirements to code through pre-trained large code models; the other is defect repair tools based on static code analysis (such as SonarQube, FindBugs, etc.), which detect syntactic or simple logical defects in the code through preset rules and provide fixed repair suggestions.
[0003] Although existing technologies have achieved basic functions such as code generation and defect repair, the following core issues still exist in practical industrial applications, making it difficult to meet enterprise-level and domain-specific R&D needs:
[0004] 1. Limited input modality and incomplete requirement analysis: Existing code generation models mostly only support natural language text input, failing to effectively handle non-textual requirement carriers such as flowcharts, architecture diagrams, and data table structures. In actual software development, requirements are often scattered across multiple types of documents. Limited input modality leads to the loss of requirement information, resulting in deviations between the generated code and the actual requirements.
[0005] 2. Poor domain adaptability and insufficient code standardization: General code generation models lack professional knowledge in vertical domains (such as finance, industrial internet, and embedded systems). The generated code often does not conform to industry standards (such as signature verification in financial transactions and memory management constraints in embedded code), requiring a lot of manual modification before it can be put into use.
[0006] 3. Rigid defect repair and lack of adaptability: Existing defect repair tools rely on preset defect templates and repair rules, and cannot dynamically adjust the repair strategy according to the root cause of the defect and the characteristics of the domain.
[0007] 4. Fragmented workflow and lack of closed-loop optimization mechanism: In the existing solution, requirement analysis, code generation, test case generation, and defect fixing are independent processes, lacking an end-to-end flow and feedback mechanism. Defect fixing results cannot optimize the code generation model, and test cases cannot accurately match the core logic of the generated code, resulting in limited improvement in overall R&D efficiency.
[0008] 5. The semantic gap between requirements and code remains unresolved: Existing technologies have not established a precise mapping relationship between requirements and code. The semantic information of natural language requirements is difficult to effectively transform into the syntax and logical features of code, resulting in problems such as logical loopholes and missing functions in the generated code. Summary of the Invention
[0009] In view of the deficiencies of the prior art mentioned in the background, the purpose of this invention is to provide a method and system for code generation and defect repair based on multimodal input.
[0010] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a code generation and defect repair method based on multimodal input, comprising:
[0011] Acquire multimodal data;
[0012] A differentiated preprocessing strategy is used to extract features from the multimodal data to generate multimodal feature vectors;
[0013] Collect vertical domain data and construct a vertical domain knowledge graph based on the vertical domain data;
[0014] Generate target code that conforms to the specifications based on the multimodal feature vectors and the vertical domain knowledge graph;
[0015] A dual detection mechanism based on AI semantic analysis and knowledge graph rule matching is used to detect and repair defects in the target code.
[0016] As a specific implementation of this application, the multimodal data includes text data, image data, and tabular data; the multimodal feature vector includes text feature vector, image feature vector, and tabular feature vector; generating the multimodal feature vector specifically involves:
[0017] For the text data, the BERT / RoBERTa model is used for word segmentation, named entity recognition, and semantic role labeling to extract core elements, which are then converted into text feature vectors. The core elements include functional requirements, non-functional requirements, and constraints.
[0018] For the image data, the YOLO model is used for object detection to extract elements from the image, and a graph neural network is used to parse the relationships between multiple elements in the image to generate image feature vectors; the elements in the image include components, interfaces, and data streams;
[0019] For tabular data, extract the field names and data types, and convert the field names and data types into table feature vectors using an encoding model.
[0020] As one specific implementation of this application, the vertical domain data includes domain technical documents, open-source code library comments, industry standard manuals, and defect repair case data; a vertical domain knowledge graph is constructed based on the vertical domain data, specifically as follows:
[0021] The BERT and ERNIE models are used to extract knowledge from the vertical domain data to obtain entities, relationships and rules.
[0022] The entities, relationships, and rules are subjected to ambiguity resolution and duplication merging to construct a vertical domain knowledge graph.
[0023] As a specific implementation of this application, target code conforming to the specifications is generated based on the multimodal feature vectors and the vertical domain knowledge graph, specifically as follows:
[0024] The text feature vector, image feature vector, and table feature vector are semantically aligned and fused to generate a unified semantic vector containing complete requirement information.
[0025] The vertical domain knowledge graph is transformed into a vector to obtain a knowledge graph vector;
[0026] The unified semantic vector is matched with the knowledge graph vector to obtain the constraints for code generation;
[0027] The constraints are converted into instruction format, the number of instructions is input into the code generation model, and target code that conforms to the specifications is generated.
[0028] As a specific implementation of this application, the defect detection and repair of the target code specifically involves:
[0029] Based on the CodeLlama / StarCoder pre-trained large code model, logical vulnerabilities and performance issues in the target code are identified through code semantic analysis;
[0030] Based on the normative rules in the vertical domain knowledge graph, it is determined whether the target code conforms to industry standards.
[0031] As another optional implementation, the defect detection and repair of the target code specifically involves:
[0032] Static analysis tools were used to perform static analysis on the target code to identify potential defect risks.
[0033] The target code is dynamically tested based on a deep learning model to capture abnormal behavior information during code execution.
[0034] The target code is semantically validated using the vertical domain knowledge graph to ensure that the code meets business requirements and programming standards.
[0035] As a preferred implementation of this application, after performing defect detection and repair on the target code, the method further includes:
[0036] The detected defects are labeled with risk levels and located, and a structured defect report is generated.
[0037] Furthermore, as a preferred implementation of this application, the method further includes adaptive repair based on the structured defect report, specifically:
[0038] By combining the defect-root cause relationships in the aforementioned vertical domain knowledge graph, the root causes of defects are analyzed.
[0039] Based on the root cause, domain rules, and defect types recorded in the structured defect report, the optimal remediation strategy is matched from the vertical domain knowledge graph;
[0040] The target code is adaptively repaired using the optimal repair strategy described above.
[0041] Furthermore, as a preferred implementation of this application, the method further includes test case generation and verification, specifically:
[0042] Test cases are automatically generated based on the target code and the requirement information in the vertical domain knowledge graph;
[0043] The test cases are automatically verified by an intelligent processing program.
[0044] Secondly, embodiments of this application also provide a code generation and defect repair system based on multimodal input, including:
[0045] The acquisition unit is used to acquire multimodal data;
[0046] A multimodal processing unit is used to extract features from the multimodal data using differentiated preprocessing strategies to generate multimodal feature vectors.
[0047] A knowledge graph construction unit is used to collect vertical domain data and construct a vertical domain knowledge graph based on the vertical domain data.
[0048] The code generation unit is used to generate target code that conforms to the specifications based on the multimodal feature vectors and the vertical domain knowledge graph;
[0049] The defect detection and repair unit is used to perform defect detection and repair on the target code using a dual detection mechanism based on AI semantic analysis and knowledge graph rule matching.
[0050] Thirdly, embodiments of the present invention also provide another code generation and defect repair system based on multimodal input, including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method described in the first aspect.
[0051] Implementing embodiments of the present invention provides an AI code generation and defect repair scheme based on multimodal input and vertical domain knowledge graphs, with the following advantages:
[0052] 1. More complete requirement analysis: Supports multimodal input of text, images, and tables. Cross-modal fusion enables complete extraction of requirement information, improving the accuracy and comprehensiveness of requirement analysis and significantly reducing the deviation between generated code and requirements.
[0053] 2. Enhanced code domain adaptability: The introduction of domain knowledge graphs ensures that the generated code conforms to industry standards and can be put into use without manual modification.
[0054] 3. More accurate and flexible defect repair: The adaptive repair engine dynamically generates repair strategies based on the root cause of defects and domain rules, improving the accuracy and flexibility of defect repair.
[0055] 4. Significantly improved R&D efficiency: The end-to-end closed-loop chain automates the entire process of "requirements → code → testing → fixes", shortening the software development cycle and reducing labor costs.
[0056] 5. The system is highly scalable: It supports custom expansion of domain knowledge graphs and incremental training of new programming languages / frameworks, and can adapt to the personalized needs of different industries, making it widely applicable. Attached Figure Description
[0057] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.
[0058] Figure 1 This is a flowchart of the code generation and defect repair method based on multimodal input provided in the embodiments of the present invention;
[0059] Figure 2 This is a structural diagram of the code generation and defect repair system based on multimodal input provided in an embodiment of the present invention;
[0060] Figure 3 yes Figure 2 Another structural diagram of the system shown. Detailed Implementation
[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0062] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0063] Please refer to Figure 1 The present invention provides a method for code generation and defect repair based on multimodal input, which may include the following steps:
[0064] S1, acquire multimodal data.
[0065] Multimodal data includes text data, image data, and tabular data.
[0066] S2, a differentiated preprocessing strategy is used to extract features from the multimodal data to generate multimodal feature vectors.
[0067] In this embodiment, differentiated preprocessing strategies are adopted for different types of input requirements (text, images, tables) to transform unstructured / semi-structured data into structured feature vectors that the model can recognize, as follows:
[0068] For text-based requirements (such as requirement documents): BERT / RoBERTa models are used for word segmentation, named entity recognition (NER), semantic role labeling (SRL), etc., to extract core elements such as functional requirements, non-functional requirements, and constraints, and generate text feature vectors.
[0069] For image-related requirements (such as flowcharts, architecture diagrams, etc.): Use the YOLO model for object detection to extract elements such as components, interfaces, and data flows from the graph; use a graph neural network (GNN) to analyze the relationships between elements and generate image feature vectors.
[0070] For tabular requirements (such as data table structure): extract field names, data types, primary key / foreign key relationships, generate a structured description in Database Modeling Language (SQL DDL), and then convert it into a table feature vector through an encoding model.
[0071] For example, in the financial industry, the text requirement is "develop a user account transfer interface", the flowchart includes data flow such as "user → transfer interface → database", and the data table structure includes a user table (user_id, name, balance) and a transfer record table (transfer_id, user_id, amount, time).
[0072] The above multimodal data is preprocessed as follows: For text requirements, elements such as "transfer interface" and "user account" are extracted using RoBERTa to generate text feature vectors; for flowcharts, elements such as "transfer interface", "user" and "database" are extracted using YOLO, and data flow relationships are parsed using GNN to generate image feature vectors; for data table structures, field relationships are extracted to generate table feature vectors.
[0073] S3. Collect vertical domain data and construct a vertical domain knowledge graph based on the vertical domain data.
[0074] The vertical domain data includes domain technical documents, open-source code library comments, industry standard manuals, and defect repair case data.
[0075] In this embodiment, a vertical domain knowledge graph with a four-layer structure of entities, relationships, attributes, and rules is constructed to provide domain knowledge support for code generation and defect repair.
[0076] Knowledge Acquisition: Crawling data such as technical documents, open-source code library comments, industry standard manuals, and defect repair cases in the field.
[0077] Knowledge extraction: A hybrid approach of "rule + deep learning" is adopted, using the BERT+CRF model to extract entities, the ERNIE model to extract relations, and the Protégé ontology modeling tool to extract rules.
[0078] Knowledge fusion: Eliminate entity ambiguity (such as the polysemy of "Java"), merge duplicate relationships, and build a unified knowledge graph; use Neo4j / JanusGraph graph database for storage and support incremental updates.
[0079] Knowledge representation: Transforming entities, relations, and rules in a knowledge graph into computable vector representations, which facilitates fusion with multimodal feature vectors.
[0080] For example, in the financial field, data is collected from the State Financial Regulatory Commission's financial code specifications, financial transaction interface development documents, and open-source financial project defect repair cases; entities such as "financial transaction interface," "SHA-256 encryption," and "idempotency verification" are extracted using BERT+CRF; relationships such as "financial transaction interface → must contain → signature verification" are extracted using ERNIE; and the constructed knowledge graph is stored in the Neo4j database, supporting incremental updates.
[0081] S4. Generate target code that conforms to the specifications based on the multimodal feature vectors and the vertical domain knowledge graph.
[0082] In practice, step S4 includes:
[0083] (1) Perform semantic alignment and fusion processing on the text feature vector, image feature vector and table feature vector to generate a unified semantic vector containing complete requirement information.
[0084] This embodiment proposes a "cross-modal attention alignment + feature weighted fusion" mechanism to achieve semantic alignment and fusion of text, image, and table features, generating a unified semantic vector containing complete requirement information, specifically including:
[0085] Semantic alignment: By calculating the association weights between features of different modalities through a cross-modal attention mechanism (such as the association degree between "user login" in text and "login component" in image), accurate matching of multimodal semantics is achieved.
[0086] Feature fusion: Based on the association weight, the feature vectors of text, image and table are weighted and fused to generate a fused semantic vector; an attention masking mechanism is introduced in the fusion process to filter invalid demand information and improve fusion efficiency.
[0087] (2) The knowledge graph of the vertical domain is vectorized to obtain the knowledge graph vector.
[0088] In this embodiment, the semantic vector is fused with entities / relationships in the knowledge graph and cosine similarity is used to extract domain knowledge related to the current requirement (e.g., if the requirement is "financial transaction interface", then the rule "financial domain → transaction interface → must include signature / log / idempotency verification" is matched).
[0089] For example, semantic vectors are fused with financial knowledge graphs for matching, extracting constraints such as "transfer interface → must include signature verification, logging, and idempotency verification" and "amount field must be transmitted encrypted." Code generation: These constraints are injected into the CodeLlama model to generate transfer interface code based on Java + Spring Boot, including SHA-256 signature logic, Slf4j logging, idempotency verification based on transfer_id, and AES encrypted transmission logic for amount, and automatically generating interface documentation.
[0090] (3) Match the unified semantic vector with the knowledge graph vector to obtain the constraints for code generation.
[0091] (4) Convert the constraints into instruction format, input the number of instructions into the code generation model, and generate target code that conforms to the specifications.
[0092] Specifically, the matched domain knowledge is transformed into constraints for code generation (such as "use SHA-256 encryption" and "prohibit plaintext storage of passwords"), and then transformed into an instruction format that the model can recognize, which is then passed to the code generation model.
[0093] S5 employs a dual detection mechanism based on AI semantic analysis and knowledge graph rule matching to detect and repair defects in the target code.
[0094] In this embodiment, a dual detection mechanism of "AI semantic analysis + knowledge graph rule matching" is adopted, covering five categories: grammatical defects, logical defects, performance defects, security defects, and specification defects.
[0095] AI Semantic Analysis: Based on pre-trained large-scale code models using CodeLlama / StarCoder, it identifies logical vulnerabilities (such as array out-of-bounds errors and null pointer references) and performance issues (such as redundant loops) through code semantic analysis. For example, AI semantic analysis can discover the logical flaw of "not performing non-negative validation on transfer amounts" in the code; and it can discover the specification flaw of "not verifying signature results" through knowledge graph rule matching, marking the risk level as "high".
[0096] Knowledge graph rule matching: Based on the normative rules in the domain knowledge graph, it detects whether the code complies with industry regulations (such as medical data processing complying with HIPAA regulations, and financial codes complying with the requirements of the State Financial Supervision and Administration Bureau).
[0097] Optionally, defect detection of the target code specifically involves:
[0098] Static analysis tools are used to perform static analysis on the target code to identify potential defects and risks. Static analysis tools such as Pylint and Mypy can check for common errors and potential security vulnerabilities in the code. In other words, static analysis refers to detecting syntax errors, type problems, and potential logical vulnerabilities in the code before compilation / running through methods such as syntax parsing, rule matching, and type inference.
[0099] The target code is dynamically tested based on a deep learning model to capture abnormal behavior information during code execution.
[0100] The target code is semantically validated using the vertical domain knowledge graph to ensure that the code meets business requirements and programming standards.
[0101] The above-mentioned code defect detection method is completed automatically by the system. To better ensure the accuracy of defect detection, this embodiment of the invention also provides a manual inspection step. Specifically, senior developers / architects manually inspect the code, considering aspects such as business rationality and business application scenarios, which can detect defects that the system has not identified.
[0102] Furthermore, after code defect detection is completed, the remediation phase will begin, which can be done in the following ways:
[0103] 1. Automated Repair: Utilizing tools such as Black, Autopep8, MonkeyType, and Snyk to directly modify code, it can automatically repair defects such as coding style violations, missing simple type annotations, code formatting issues, and basic security vulnerabilities. Furthermore, this automated repair method is suitable for scenarios such as real-time repair during the development phase.
[0104] 2. Manual Fix: Developers manually modify the code based on the defect detection results. This method is suitable for fixing complex defects. Complex defects include, but are not limited to, fatal / critical defects, complex business logic vulnerabilities, and architectural defects.
[0105] S6 identifies and locates the risk levels of detected defects and generates a structured defect report.
[0106] Specifically, the detected defects are labeled with risk level (high / medium / low) and precisely located (file path, line number, defect type) to generate a structured defect report.
[0107] S7, perform adaptive repair based on the structured defect report.
[0108] In this embodiment, step S7 includes:
[0109] By combining the defect-root cause relationships in the aforementioned vertical domain knowledge graph, the root causes of defects are analyzed.
[0110] Based on the root cause, domain rules, and defect types recorded in the structured defect report, the optimal remediation strategy is matched from the vertical domain knowledge graph. Taking the financial domain as an example, the optimal remediation strategy is to add non-negative amount verification logic (if (amount<0) throw new IllegalArgumentException("transfer amount cannot be negative")) and add signature verification logic (based on SHA-256 public key verification).
[0111] The target code is adaptively repaired using the optimal repair strategy described above.
[0112] S8, Test Case Generation and Verification.
[0113] In this embodiment, step S8 includes:
[0114] Test cases are automatically generated based on the target code and the requirement information in the vertical domain knowledge graph;
[0115] The test cases are automatically verified by an intelligent processing program.
[0116] The AI code generation and defect repair scheme based on multimodal input and vertical domain knowledge graph provided in this invention first acquires multimodal data, then extracts features using a differentiated preprocessing strategy to generate multimodal feature vectors, constructs a vertical domain knowledge graph, generates compliant target code based on the multimodal feature vectors and the vertical domain knowledge graph, and finally uses a dual detection mechanism of AI semantic analysis + knowledge graph rule matching to detect and repair defects in the target code. Compared with the prior art, the advantages of this invention are as follows:
[0117] 1. More complete requirement analysis: Supports multimodal input of text, images, and tables. Cross-modal fusion enables complete extraction of requirement information, improving the accuracy and comprehensiveness of requirement analysis and significantly reducing the deviation between generated code and requirements.
[0118] 2. Enhanced code domain adaptability: The introduction of domain knowledge graphs ensures that the generated code conforms to industry standards and can be put into use without manual modification.
[0119] 3. More accurate and flexible defect repair: The adaptive repair engine dynamically generates repair strategies based on the root cause of defects and domain rules, improving the accuracy and flexibility of defect repair.
[0120] 4. Significantly improved R&D efficiency: The end-to-end closed-loop chain automates the entire process of "requirements → code → testing → fixes", shortening the software development cycle and reducing labor costs.
[0121] 5. The system is highly scalable: It supports custom expansion of domain knowledge graphs and incremental training of new programming languages / frameworks, and can adapt to the personalized needs of different industries, making it widely applicable.
[0122] Based on the same inventive concept, embodiments of the present invention provide a code generation and defect repair system based on multimodal input, such as... Figure 2 As shown, it includes:
[0123] The acquisition unit is used to acquire multimodal data;
[0124] A multimodal processing unit is used to extract features from the multimodal data using differentiated preprocessing strategies to generate multimodal feature vectors.
[0125] A knowledge graph construction unit is used to collect vertical domain data and construct a vertical domain knowledge graph based on the vertical domain data.
[0126] The code generation unit is used to generate target code that conforms to the specifications based on the multimodal feature vectors and the vertical domain knowledge graph;
[0127] The defect detection and repair unit is used to perform defect detection and repair on the target code using a dual detection mechanism based on AI semantic analysis and knowledge graph rule matching.
[0128] As a specific implementation, the multimodal data includes text data, image data, and tabular data; the multimodal feature vector includes text feature vectors, image feature vectors, and tabular feature vectors; the multimodal processing unit shown is specifically used for:
[0129] For the text data, the BERT / RoBERTa model is used for word segmentation, named entity recognition, and semantic role labeling to extract core elements, which are then converted into text feature vectors. The core elements include functional requirements, non-functional requirements, and constraints.
[0130] For the image data, the YOLO model is used for object detection to extract elements from the image, and a graph neural network is used to parse the relationships between multiple elements in the image to generate image feature vectors; the elements in the image include components, interfaces, and data streams;
[0131] For tabular data, extract the field names and data types, and convert the field names and data types into table feature vectors using an encoding model.
[0132] As a specific implementation, the vertical domain data includes domain technical documents, open-source code library comments, industry standard manuals, and defect repair case data; the knowledge graph construction unit is specifically used for:
[0133] The BERT and ERNIE models are used to extract knowledge from the vertical domain data to obtain entities, relationships and rules.
[0134] The entities, relationships, and rules are subjected to ambiguity resolution and duplication merging to construct a vertical domain knowledge graph.
[0135] As one specific implementation, the code generation unit is specifically used for:
[0136] The text feature vector, image feature vector, and table feature vector are semantically aligned and fused to generate a unified semantic vector containing complete requirement information.
[0137] The vertical domain knowledge graph is transformed into a vector to obtain a knowledge graph vector;
[0138] The unified semantic vector is matched with the knowledge graph vector to obtain the constraints for code generation;
[0139] The constraints are converted into instruction format, the number of instructions is input into the code generation model, and target code that conforms to the specifications is generated.
[0140] As one specific implementation, the defect detection and repair unit is specifically used for:
[0141] Based on the CodeLlama / StarCoder pre-trained large code model, logical vulnerabilities and performance issues in the target code are identified through code semantic analysis;
[0142] Based on the normative rules in the vertical domain knowledge graph, it is determined whether the target code conforms to industry standards.
[0143] Alternatively, as another specific implementation, the defect detection and repair unit is specifically used for:
[0144] Static analysis tools were used to perform static analysis on the target code to identify potential defect risks.
[0145] The target code is dynamically tested based on a deep learning model to capture abnormal behavior information during code execution.
[0146] The target code is semantically validated using the vertical domain knowledge graph to ensure that the code meets business requirements and programming standards.
[0147] As a preferred implementation of this application, the system further includes:
[0148] The defect report generation unit is used to label and locate the risk level of detected defects and generate a structured defect report.
[0149] As a preferred implementation of this application, the system further includes a test case generation and verification unit, used for:
[0150] Test cases are automatically generated based on the target code and the requirement information in the vertical domain knowledge graph;
[0151] The test cases are automatically verified by an intelligent processing program.
[0152] It should be noted that the specific workflow of this embodiment is described in the foregoing method embodiment section, and will not be repeated here.
[0153] Furthermore, such as Figure 3 As shown, another embodiment of the present invention also provides a code generation and defect repair system based on multimodal input, which may include: one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104. The processors 101, input devices 102, output devices 103, and memory 104 are interconnected via a bus 105. The memory 104 is used to store a computer program, the computer program including program instructions, and the processor 101 is configured to invoke the program instructions to execute the method described in the above-described method embodiment.
[0154] It should be understood that, in this embodiment of the invention, the processor 101 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0155] Input device 102 may include a keyboard, etc., and output device 103 may include a display (LCD, etc.), a speaker, etc.
[0156] The memory 104 may include read-only memory and random access memory, and provides instructions and data to the processor 101. A portion of the memory 104 may also include non-volatile random access memory. For example, the memory 104 may also store device type information.
[0157] In specific implementations, the processor 101, input device 102, and output device 103 described in the embodiments of the present invention can execute the implementation methods described in the embodiments of the code generation and defect repair method based on multimodal input provided in the embodiments of the present invention, which will not be repeated here.
[0158] Accordingly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, implement the above-described code generation and defect repair method based on multimodal input.
[0159] The computer-readable storage medium can be an internal storage unit of the system described in any of the foregoing embodiments, such as the system's hard disk or memory. The computer-readable storage medium can also be an external storage device of the system, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the system. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.
[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0161] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.
[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.
[0163] Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units. When using each module, user information is collected and stored only with the user's full authorization and in compliance with relevant laws and regulations, protecting the security and privacy of user data, and strictly prohibiting unauthorized access; data processing will be conducted within the scope stipulated by law and will not exceed the purpose and scope authorized by the user; at the same time, users have the rights to access, correct, delete, restrict processing, and refuse their personal data; and must strictly comply with applicable laws and regulations and conduct compliance reviews.
[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0165] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for code generation and defect repair based on multimodal input, characterized in that, include: Acquire multimodal data; A differentiated preprocessing strategy is used to extract features from the multimodal data to generate multimodal feature vectors; Collect vertical domain data and construct a vertical domain knowledge graph based on the vertical domain data; Generate target code that conforms to the specifications based on the multimodal feature vectors and the vertical domain knowledge graph; A dual detection mechanism based on AI semantic analysis and knowledge graph rule matching is used to detect and repair defects in the target code.
2. The method as described in claim 1, characterized in that, The multimodal data includes text data, image data, and tabular data; the multimodal feature vectors include text feature vectors, image feature vectors, and tabular feature vectors; generating multimodal feature vectors specifically involves: For the text data, the BERT / RoBERTa model is used for word segmentation, named entity recognition, and semantic role labeling to extract core elements, which are then converted into text feature vectors. The core elements include functional requirements, non-functional requirements, and constraints. For the image data, the YOLO model is used for object detection to extract elements from the image, and a graph neural network is used to parse the relationships between multiple elements in the image to generate image feature vectors; the elements in the image include components, interfaces, and data streams; For tabular data, extract the field names and data types, and convert the field names and data types into table feature vectors using an encoding model.
3. The method as described in claim 2, characterized in that, The vertical domain data includes domain technical documents, open-source code library comments, industry standard manuals, and defect repair case data; a vertical domain knowledge graph is constructed based on the vertical domain data, specifically as follows: The BERT and ERNIE models are used to extract knowledge from the vertical domain data to obtain entities, relationships and rules. The entities, relationships, and rules are subjected to ambiguity resolution and duplication merging to construct a vertical domain knowledge graph.
4. The method as described in claim 3, characterized in that, Based on the multimodal feature vectors and the vertical domain knowledge graph, target code conforming to the specifications is generated, specifically as follows: The text feature vector, image feature vector, and table feature vector are semantically aligned and fused to generate a unified semantic vector containing complete requirement information. The vertical domain knowledge graph is transformed into a vector to obtain a knowledge graph vector; The unified semantic vector is matched with the knowledge graph vector to obtain the constraints for code generation; The constraints are converted into instruction format, the number of instructions is input into the code generation model, and target code that conforms to the specifications is generated.
5. The method as described in claim 1, characterized in that, The specific steps for defect detection in the target code are as follows: Based on the CodeLlama / StarCoder pre-trained large code model, logical vulnerabilities and performance issues in the target code are identified through code semantic analysis; Based on the normative rules in the vertical domain knowledge graph, it is determined whether the target code conforms to industry standards.
6. The method as described in claim 1, characterized in that, The specific steps for defect detection in the target code are as follows: Static analysis tools were used to perform static analysis on the target code to identify potential defect risks. The target code is dynamically tested based on a deep learning model to capture abnormal behavior information during code execution. The target code is semantically validated using the vertical domain knowledge graph to ensure that the code meets business requirements and programming standards.
7. The method as described in claim 5, characterized in that, After performing defect detection and repair on the target code, the method further includes: The detected defects are labeled with risk levels and located, and a structured defect report is generated.
8. The method as described in claim 7, characterized in that, The method further includes adaptive repair based on the structured defect report, specifically: By combining the defect-root cause relationships in the aforementioned vertical domain knowledge graph, the root causes of defects are analyzed. Based on the root cause, domain rules, and defect types recorded in the structured defect report, the optimal remediation strategy is matched from the vertical domain knowledge graph; The target code is adaptively repaired using the optimal repair strategy described above.
9. The method as described in claim 8, characterized in that, The method also includes test case generation and verification, specifically: Test cases are automatically generated based on the target code and the requirement information in the vertical domain knowledge graph; The test cases are automatically verified by an intelligent processing program.
10. A code generation and defect repair system based on multimodal input, characterized in that, include: The acquisition unit is used to acquire multimodal data; A multimodal processing unit is used to extract features from the multimodal data using differentiated preprocessing strategies to generate multimodal feature vectors. A knowledge graph construction unit is used to collect vertical domain data and construct a vertical domain knowledge graph based on the vertical domain data. The code generation unit is used to generate target code that conforms to the specifications based on the multimodal feature vectors and the vertical domain knowledge graph; The defect detection and repair unit is used to perform defect detection and repair on the target code using a dual detection mechanism based on AI semantic analysis and knowledge graph rule matching.
11. A code generation and defect repair system based on multimodal input, characterized in that, The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to perform the method as described in any one of claims 1-9.