A multi-index scalable multi-agent code generation evaluation system and method

Through the multi-module collaborative work of multi-agent systems, the problems of manual error, engineering bottleneck and poor dynamic adaptability in the code generation evaluation of large language model are solved, and efficient and flexible multi-dimensional evaluation is achieved, which improves the accuracy of evaluation. and adaptability.

CN118838629BActive Publication Date: 2024-12-06NANJING NEW GENERATION ARTIFICIAL INTELLIGENCE RES INST CO LTD +2
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411324875.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2024-12-06
Estimated Expiration
2044-09-23

AI Technical Summary

Technical Problem

The existing large language model code generation evaluation methods have problems such as manual evaluation error, engineering technology bottlenecks, single evaluation dimensions and poor dynamic adaptability, and it is difficult to comprehensively, flexibly and efficiently evaluate code generation capabilities.

Method used

Adopt multi-agent system, including interactive multi-agent modules, verification agent modules and management agent modules, forms a closed loop through mutual communication and feedback between modules, realizes multi-dimensional evaluation and dynamic optimization, and integrates multiple evaluation indicators and data processing. Policy, supports continuous updates and expansion of the system.

Benefits of technology

Reduce manual intervention, improve evaluation efficiency and accuracy, enhance system flexibility and adaptability, be able to consider multi-dimensional indicators generated by code at the same time, adapt to model updates and environmental changes, and ensure the forward-looking and robustness of evaluation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118838629B_ABST
    Figure CN118838629B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-indicator scalable multi-agent code generation evaluation system and method, belonging to the technical field of large language model (LLM) evaluation. The evaluation system comprises an interactive multi-agent module, a verification agent module, and a management agent module. Each agent in the interactive multi-agent module processes its specific API call mechanism, parameter configuration, data format and error handling mechanism for different large language models, inputs an evaluation benchmark data set, and realizes fast calling and synchronous evaluation; the verification agent module integrates multiple evaluation benchmarks, performs multi-dimensional evaluation of code correctness, efficiency and readability, and comprehensively compares and evaluates code generation indicators of different large language models; the management agent module is responsible for overall scheduling, guiding a single agent to automatically generate and optimize a data set according to evaluation feedback, thereby realizing continuous optimization of autonomous agents and data sets.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence (AI) and machine learning (ML), and in particular to an evaluation technology of a large language model (LLM). Background Art

[0002] Code generation is an automated program development technology that aims to generate code based on given specifications and constraints; the large language model evaluation method is generally used to comprehensively evaluate the capabilities of the large language model based on the corresponding evaluation benchmark of the large language model's functions, to discover loopholes and deficiencies in the model, and to provide a reference for model improvement.

[0003] The main technologies currently used in the industry are: Traditional code generation methods: mainly rely on high-quality vocabularies, manually constructed templates and language features in specific fields, and manually write logical rules to generate code snippets. This method has great limitations and is difficult to adapt to complex and changing programming environments, while increasing the writing burden on developers.

[0004] Deep learning code generation methods: mainly divided into three categories: methods based on code features, methods combined with retrieval, and methods combined with post-processing.

[0005] Existing code generation evaluation methods: Currently, the quality evaluation of large language model code generation mainly relies on automated code evaluation benchmark indicators such as APPS, RoboCodeGen, CodeScore-R and HumanEval.

[0006] Existing problems: Considering the existing LLM evaluation method and intelligent system model, there are four problems: manual evaluation error, engineering technology bottleneck, single evaluation dimension and poor dynamic adaptability:

[0007] Manual evaluation errors: Manual evaluation is subjective, cannot guarantee accuracy and quality, and is inefficient.

[0008] Engineering bottlenecks: Each large language model has different API (Application Programming Interface) calling logic, parameter configuration (such as API key, endpoint URL, etc.), data format (such as JSON, XML, etc.), error handling mechanism and security authentication mechanism (such as API key, OAuth2, etc.). When evaluating and comparing multiple large language models at the same time, it is difficult to switch quickly and flexibly, and it is not convenient for subsequent maintenance.

[0009] Single evaluation dimension: Most large language model code generation evaluation methods only focus on single-dimensional performance, such as code correctness or task success rate, and fail to consider multiple indicators such as code generation correctness, efficiency, and readability, resulting in one-sided evaluation results and lack of robustness. For example, the APPS benchmark mainly focuses on whether the code generated by the model can pass unit testing, ignoring the readability and robustness of the code.

[0010] Poor dynamic adaptability: Using predefined static data sets lacks dynamic adjustment and iterative optimization mechanisms, making it difficult to cope with model updates and environmental changes. For example, existing code generation benchmarks are difficult to maintain the foresight and adaptability of their evaluations when faced with constantly updated code bases and APIs.

[0011] Multi-agent system is a relatively mature application in the field of distributed artificial intelligence. Its advantage is that it can decompose complex AI problems into multiple sub-problems, which can be solved by different agents in collaboration, thereby reducing complexity and improving resource utilization efficiency. In recent years, with the rise of large language models, code generation evaluation has also been widely discussed in the field of artificial intelligence large language models (LLMs), but how to effectively combine multi-agent systems with large language model code generation evaluation is an issue that needs to be solved in the industry. Summary of the invention

[0012] In view of the problems existing in the above-mentioned prior art, the purpose of the present invention is to effectively combine the multi-agent system with the code generation evaluation of a large language model, and to provide a comprehensive, flexible and efficient evaluation system and evaluation method for the code generation capability of a large language model.

[0013] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows: the present invention introduces a multi-agent code generation evaluation system integrating three modules, including an interactive multi-agent module, a verification agent module and a management agent module; first, the interactive multi-agent module assumes the responsibilities of large language model configuration and call, data set optimization and supplement, data format standardization processing, etc.; the verification agent module assumes the responsibilities of multi-dimensional test evaluation, visualization result generation and result feedback, etc.; and the management agent module is responsible for receiving external instructions, system deployment, data processing, supervision feedback, etc. The three modules communicate with each other and complete the evaluation task autonomously throughout the process. Secondly, the multi-agents communicate and feedback with each other, and iteratively optimize and supplement the training data set according to the evaluation results to form a closed loop. Thirdly, the verification agent module can integrate multiple evaluation indicators at the same time to ensure the consistency of the evaluation standards. Finally, each agent in the multi-agent system is independently updated and optimized, and agents can be dynamically increased or decreased according to demand.

[0014] As a preferred embodiment of the present invention, the management agent module includes a knowledge base, a database, a feature engineering and a strategy library;

[0015] The knowledge base is used to store knowledge in the field of code generation and evaluation, as well as API documentation and usage examples of various large language models, and is regularly updated to include the latest technology trends and programming paradigms; wherein, knowledge in the field of evaluation includes: various code evaluation benchmarks, code testing theories, programming language feature knowledge, and specific coding specifications and quality requirements for various fields (such as Web development, mobile applications, embedded systems, etc.).

[0016] The database is used for trend analysis and model performance comparison. The database stores multiple benchmark data sets, includes code example libraries for different programming languages ​​and task types, and can maintain historical evaluation data.

[0017] The feature engineering: according to different test tasks, model performance requirements and previous evaluation result feedback, the raw data is processed and converted to extract features with high correlation or importance with the target variable, and the evaluation data set is optimized and input into the interactive multi-agent module for training;

[0018] The strategy library is used to store optimization strategies for evaluation indicators, dynamic weight adjustment strategies for evaluation indicators, strategies for data set optimization, strategies for intelligent agent collaboration and task allocation, and strategies for system self-diagnosis and performance tuning.

[0019] As a preferred embodiment of the present invention, the interactive multi-agent module includes a data set training module, a large language model interaction module, a data conversion module and an autonomous agent module;

[0020] The data set training module is used to input multiple benchmark data sets that have been processed by feature engineering in the management agent module and are specifically targeted at code generation for training;

[0021] The large language model interaction module includes multiple different agents, each of which initializes the API client and configuration according to the assigned large language model and the corresponding API key, communicates with multiple large language models through different API call mechanisms, and generates code results;

[0022] The data conversion module: performs internal standardization conversion on the data format differences of codes generated by different agents, and inputs the data to the verification agent module;

[0023] The autonomous agent module: automatically generates more new samples under the original sample training of the benchmark data set according to the feedback of the evaluation results of the management intelligent agent module to form a supplementary data set.

[0024] As a preferred embodiment of the present invention, the verification agent module includes a benchmark evaluation index module and a verification agent;

[0025] The benchmark evaluation index module integrates multiple benchmark evaluation indicators to comprehensively evaluate the results of code generation of each model;

[0026] The verification agent: performs a multi-dimensional evaluation of the code, quantifies and comprehensively calculates the scores of each dimension according to preset weights, generates visual evaluation results, and feeds back the evaluation results to the management agent module.

[0027] In order to achieve the above object, the present invention also provides a multi-index scalable multi-agent code generation evaluation method, comprising the following steps:

[0028] Step 1: The management agent module performs system initialization and task allocation;

[0029] Step 2: The interactive multi-agent module receives the data set and code generation task processed by the management agent module. Each agent communicates with multiple large language models through different API call mechanisms to generate code results.

[0030] Step 3: The verification agent module receives the code results in a standardized format from the interactive multi-agent module and performs multi-dimensional evaluation calculations to generate an evaluation report;

[0031] Step 4: The management agent module receives the evaluation report from the verification agent module, diagnoses, processes and optimizes the evaluation report, and then redistributes it to the interactive multi-agent module to start a new round of evaluation cycle.

[0032] In order to solve the problem of manual evaluation errors, the present invention reduces manual operation and intervention through an autonomous agent evaluation process, avoids subjectivity and inconsistency, and allows multiple agents to collaborate and process different tasks simultaneously, greatly improving evaluation speed and efficiency.

[0033] In order to solve the bottleneck problem of engineering technology, a multi-agent system is introduced. Each agent handles the calling logic, parameter configuration, data format and error handling mechanism of its specific API for different large language models. After integration, it can be called quickly and switched flexibly, and can also be continuously expanded and integrated with new large language models.

[0034] To solve the problem of single evaluation dimension, firstly, the training agent integrates multiple evaluation benchmarks to conduct multi-dimensional evaluation, such as code correctness, efficiency, readability, etc.; secondly, multiple agents cooperate to comprehensively compare and evaluate the code generation indicators of different large language models, and select the best one according to the task.

[0035] To solve the problem of poor dynamic adaptability, firstly, a single agent can automatically generate and optimize data sets based on evaluation feedback, implement autonomous agency, supplement data sets, and automatically generate more new sample mechanisms; secondly, the system can flexibly increase or decrease agents to adapt to the access of new large language models or the elimination of old models, ensuring the continuous updating and optimization of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a schematic diagram of the structure of a multi-indicator scalable multi-agent code generation evaluation system in this embodiment.

[0037] Figure 2 Schematic diagram of the flow of a multi-indicator scalable multi-agent code generation evaluation method in this embodiment. DETAILED DESCRIPTION

[0038] In order to better understand the present invention, the technical solution of the present invention is further explained below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0039] like Figure 1 As shown, a schematic diagram of the structure of a multi-index scalable multi-agent code generation evaluation system in this embodiment is mainly divided into three parts: an interactive multi-agent module, a verification agent module, and a management agent module. The modules communicate with each other and operate in a division of labor. Each agent in the interactive multi-agent module processes its specific API call mechanism, parameter configuration, data format, and error handling mechanism for different large language models, inputs the evaluation benchmark data set, and realizes fast calling and synchronous evaluation. Through the autonomous agent evaluation process, manual operation and intervention are reduced to avoid subjectivity and inconsistency. The verification agent module integrates multiple evaluation benchmarks, performs multi-dimensional evaluations such as code correctness, efficiency, and readability, and comprehensively compares and evaluates different large language model code generation indicators. The management agent module is responsible for overall scheduling, guiding a single agent to automatically generate and optimize the data set according to the evaluation feedback, automatically generate more new samples, and can flexibly increase or decrease agents to adapt to the access of new large language models or the elimination of old models, ensuring continuous updating and optimization of the system.

[0040] 1. Interactive multi-agent module: The specific functions are as follows:

[0041] 1. Dataset training: Input multiple benchmark datasets specifically for code generation that have been processed by the management agent module feature engineering for training;

[0042] 2. Interact with large language models: Communicate with multiple large language models (such as GPT-3.5, GPT-4, Google Bard, LLAMA, Hugging Face, etc.) through different API call mechanisms to generate code results;

[0043] 3. Standardization of data output format: internal standardization conversion of data format differences of codes generated by different platforms is performed and input into the "Verification Agent Module";

[0044] 4. Agent scalability: The module can flexibly add or reduce agents to expand access to new large language models and eliminate old large language models;

[0045] 5. Autonomous agent: Based on the feedback from the evaluation results of the management agent module, more new samples are automatically generated under the original sample training of the benchmark dataset to form a supplementary dataset.

[0046] 2. Verification Agent Module: It includes a dedicated verification agent with the following functions:

[0047] 1. Benchmark evaluation indicators: Integrate multiple benchmark evaluation indicators to comprehensively evaluate the results of code generation for each model;

[0048] 2. Code generation evaluation: Generate visual evaluation results and feed the evaluation results back to the management "agent module";

[0049] 3. Continue to dynamically expand and integrate new evaluation indicators.

[0050] 3. Management Agent Module: Responsible for the coordination and management of the entire system. Its functions include:

[0051] 1. Overall scheduling: Receive task instructions, be responsible for the overall scheduling and management of the system, optimize task allocation, select evaluation benchmarks, output evaluation results, and perform system diagnosis and fault recovery.

[0052] 2. Knowledge base: stores domain knowledge related to code generation and evaluation, as well as API documentation and usage examples of various models, and is regularly updated to include the latest technology trends and programming paradigms.

[0053] 3. Database: Stores multiple benchmark datasets, including but not limited to APPS, HumanEval, CodeScore-R, etc., contains code sample libraries for different programming languages ​​and task types, and maintains historical evaluation data for trend analysis and model performance comparison.

[0054] 4. Feature Engineering: Based on different test tasks, model performance requirements, and previous evaluation feedback, the raw data is processed and transformed to extract features that are highly correlated with or important to the target variable, optimize the evaluation data set, and input it into the "Interactive Multi-Agent Module" for training.

[0055] 5. Strategy library: stores optimization strategies for evaluation indicators, dynamic weight adjustment strategies for evaluation indicators, data set optimization strategies (such as synonym replacement, grammatical structure adjustment, etc.), intelligent agent collaboration and task allocation strategies (such as load balancing), system self-diagnosis and performance tuning strategies, etc.

[0056] 6. Feedback mechanism: Accepts and verifies the evaluation results of the agent module and optimizes and adjusts the interactive multi-agent system based on these results.

[0057] In this embodiment, the well-designed interaction mechanism works together to ensure the independence and fault isolation of each module and a single agent within the module, improve the overall stability and reliability, and realize an efficient and flexible code generation evaluation system:

[0058] (1) An asynchronous communication protocol based on message passing is adopted among multiple agents, including:

[0059] Message format: Use the unified JSON format to encapsulate the message, including the message type, sender, receiver, timestamp, and payload data.

[0060] Message queue: Use a distributed message queue system (such as Apache Kafka) to manage message transmission between agents and reduce direct dependencies between agents.

[0061] Publish / Subscribe mode: Agents can subscribe to specific types of messages to achieve flexible many-to-many communication.

[0062] (2) Multi-agent design of load balancing and fault tolerance mechanisms, including:

[0063] Resource monitoring: monitor system resource usage in real time and optimize task allocation.

[0064] Elastic expansion: supports dynamic increase or decrease in the number of agents to adapt to load changes.

[0065] Independent deployment: Supports independent version control and deployment of agents. Updates to a single agent do not affect other agents.

[0066] In this embodiment, the verification agent module integrates multiple evaluation benchmarks to comprehensively evaluate the code generation performance of a large language model. The evaluation methods include:

[0067] 1. Code generation evaluation indicators and their calculation methods:

[0068] (1) Pass@k: Measures the probability that at least one of the k code samples generated by the model passes all test cases.

[0069] ,

[0070] Where: —The number of samples that pass all test cases; —Total number of samples.

[0071] (2) BLEU: Evaluates the similarity between the generated code and the reference code:

[0072] ,

[0073] Where: —reference length; - candidate length; —The precision of the i-gram.

[0074] (3) CodeBLEU: A BLEU variant designed specifically for code evaluation that takes into account abstract syntax trees and data flows:

[0075] ,

[0076] Where: —weighted n-gram match; —weighted tree matching; —Dataflow match.

[0077] (4) Perplexity: Measures the uncertainty of the model's prediction of the code:

[0078] ,

[0079] Where: — cross entropy loss;

[0080] (5) Cyclomatic Complexity: Evaluate the complexity of the code:

[0081] ,

[0082] Where: — number of sides; — Number of nodes; —Number of connected components.

[0083] (6) Halstead Metrics: Calculates program length, vocabulary, difficulty, etc. based on the number of operators and operands to measure the readability and maintainability of the code.

[0084] 2. Multi-dimensional indicator stacking method: For multiple evaluation indicators such as functional correctness, code quality, code complexity, readability and maintainability, security and efficiency, a dynamic weight calculation method is adopted to adjust the weight of each indicator according to the specific needs and importance of the task.

[0085] (1) Dynamic weight calculation formula:

[0086] ,

[0087] Where: —weight of indicator i; —Normalized score of indicator i.

[0088] (2) Dynamic weight assignment factors: task type, domain characteristics, user preferences, and historical evaluation data, etc.

[0089] In this embodiment, the code generation evaluation system is provided with a feedback optimization mechanism: the three modules of the multi-agent system communicate with each other, and the management agent module is deployed as a whole to form a complete feedback optimization mechanism within the system, which is specifically embodied in two closed loops:

[0090] 1. Small closed loop - evaluation benchmark optimization:

[0091] The evaluation results of the large language model generation effect are closely related to the command prompt in the evaluation data set. The management agent module collects the evaluation results, optimizes the benchmark data set used to train the interactive agent module by replacing synonyms, grammatical structure, and similar sentence construction, guides it to adjust the prompt instructions for interacting with the large language model, and guides the verification agent module to update the evaluation indicators.

[0092] 2. Large closed loop - data set improvement and large language model optimization:

[0093] The significance of the evaluation is to guide the continuous correction and improvement of the large language model in a more scientific and targeted manner. Based on the evaluation results, the management agent module can use the external knowledge base, strengthen the characteristics of the data set through feature engineering, encrypt and desensitize the data set, generate a supplementary data set, and fine-tune the large language model through the interactive agent module to improve its code generation capabilities.

[0094] like Figure 2 As shown, a schematic diagram of a multi-index scalable multi-agent code generation evaluation method in this embodiment includes the following steps:

[0095] Step 1: Initialization and task allocation

[0096] Step 1.1: The management agent module receives external evaluation task instructions;

[0097] Step 1.2: The management agent module selects appropriate benchmark datasets from the knowledge base and database according to task requirements;

[0098] Step 1.3: The management agent module performs feature engineering on the selected dataset;

[0099] Step 1.4: The management agent module distributes the processed data set to the interactive multi-agent module.

[0100] Step 2: Code Generation

[0101] Step 2.1: Each agent in the interactive multi-agent module receives the data set processed by the management agent module and the code generation task, which usually includes programming requirements or problem statements described in natural language;

[0102] Step 2.2: Each agent initializes the API client and configuration according to the assigned large language model (such as GPT-3.5, Google Bard, HuggingFace, etc.) and the corresponding API key;

[0103] Step 2.3: The agent converts the task into the format required by the corresponding LLM API, which involves setting specific parameters such as temperature (controlling output randomness), maximum number of tokens, etc.

[0104] Step 2.4: The agent uses the configured API client to send a request to the corresponding LLM. This involves network communication, handling timeouts, retries, etc.

[0105] Step 2.5: The agent receives and parses the generated code returned by LLM, which involves JSON parsing, error handling, etc.

[0106] Step 2.6: The agent performs format standardization on the generated code.

[0107] Step 3: Code Evaluation

[0108] Step 3.1: The verification agent module receives the generated code in a standardized format from the interactive multi-agent module;

[0109] Step 3.2: The verification agent evaluates the code in multiple dimensions, including correctness, code quality, complexity, readability, etc.

[0110] Step 3.3: The verification agent quantifies and comprehensively calculates the scores of each dimension according to the preset weights;

[0111] Step 3.4: The verification agent generates a detailed report based on the evaluation results, including scores for each dimension, identification of specific issues, suggestions for improvement, etc.

[0112] Step 4: Feedback and Optimization

[0113] Step 4.1: The management agent module receives the evaluation report of the verification agent;

[0114] Step 4.2: The management agent module uses the strategy library to diagnose and handle problems based on the evaluation results;

[0115] Step 4.3: The management agent module optimizes the data set based on the diagnosis results;

[0116] Step 4.4: The management agent module redistributes the optimized data set to the interactive multi-agent module, starting a new round of evaluation cycle.

[0117] Finally, the entire system is capable of continuous monitoring and adjustment: the management agent module dynamically adjusts parameters such as the number of agents and evaluation indicator weights as needed, regularly updates the knowledge base, database and strategy library, and dynamically expands and integrates new evaluation indicators to adapt to new evaluation needs and technological developments.

[0118] The above technical solution disclosed in this embodiment has the following advantages:

[0119] (1) Efficient multi-agent collaboration mechanism:

[0120] The system consists of interactive multi-agent modules, verification agent modules and management agent modules. Each agent module undertakes specific tasks, achieving high modularity.

[0121] It adopts an asynchronous communication protocol based on message passing, encapsulates messages in JSON format, and implements flexible many-to-many communication through a distributed message queue system.

[0122] The API calling mechanism solves the problem of unifying interfaces of different large language models, improves the scalability and adaptability of the system, and simplifies the maintenance and update process.

[0123] A complete fault-tolerant mechanism has been designed to ensure the stability and reliability of the system.

[0124] (2) Dynamic data set optimization:

[0125] Based on the evaluation feedback, the system's dynamic adaptability and continuous optimization are ensured through automatic iterative optimization of small closed loops (evaluation benchmark optimization) and large closed loops (dataset improvement and large language model optimization).

[0126] The data set is encrypted and desensitized to ensure data security and privacy protection

[0127] (3) Multi-dimensional comprehensive evaluation:

[0128] Integrates multiple evaluation indicators, including Pass@k, BLEU, CodeBLEU, Perplexity, CyclomaticComplexity and Halstead Metrics.

[0129] A dynamic weight calculation method is used to adjust the weight of each indicator according to task requirements and importance.

[0130] Multi-dimensional indicators stacked, covering functional correctness, code quality, complexity, readability, maintainability, security and efficiency

[0131] Generates detailed visual evaluation reports and supports comparative analysis of evaluation results of different large language models.

[0132] (4) System flexibility and scalability:

[0133] The system modular architecture is independently packaged to achieve independent development, testing, deployment and update

[0134] Implement agent pool management mechanism to support dynamic addition or removal of agents. Adopt distributed storage system to support horizontal expansion of data volume.

[0135] The API calling mechanism solves the problem of unifying interfaces of different large language models, improves the scalability and adaptability of the system, and simplifies the maintenance and update process.

[0136] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-index scalable multi-agent code generation evaluation system, characterized by: It includes an interactive multi-agent module, a verification agent module and a management agent module. Each agent in the interactive multi-agent module processes its corresponding API call mechanism, parameter configuration, data format and error handling mechanism for different large language models, inputs the evaluation benchmark data set, and realizes fast calling and synchronous evaluation; the verification agent module integrates multiple evaluation benchmarks, conducts multi-dimensional evaluation of code correctness, efficiency and readability, and comprehensively compares and evaluates code generation indicators of different large language models; the management agent module is responsible for overall scheduling, guiding individual agents to automatically generate and optimize data sets according to evaluation feedback, thereby realizing continuous optimization of autonomous agents and data sets; The verification agent module includes a benchmark evaluation index module and a verification agent; The benchmark evaluation index module integrates multiple benchmark evaluation indicators to comprehensively evaluate the results of code generation of each model; The verification agent: performs multi-dimensional evaluation on the code, quantifies and comprehensively calculates the scores of each dimension according to preset weights, generates visual evaluation results, and feeds back the evaluation results to the management agent module; The benchmark evaluation metrics include: Pass@k: measures the probability that at least one of the k code samples generated by the model passes all test cases. The calculation formula is: , Where: —The number of samples that pass all test cases; —Total number of samples; BLEU: Evaluates the similarity between the generated code and the reference code. The calculation formula is: , Where: —reference length; - candidate length; The precision of the i-gram; n is the order of the largest n-gram; CodeBLEU: A BLEU variant designed specifically for code evaluation, taking into account abstract syntax trees and data flows, with the calculation formula being: , Where: —weighted n-gram match; —weighted tree matching weightedtree match; —Dataflow match dataflow match; Perplexity: Measures the uncertainty of the model's prediction of the code, calculated as: , Where: — Cross entropy loss; Cyclomatic Complexity: Evaluates the complexity of the code, calculated as: , Where: — number of sides; — Number of nodes; —Number of connected components; Halstead Metrics: Calculates program length, vocabulary, and difficulty based on the number of operators and operands, measuring the readability and maintainability of code.

2. The multi-index scalable multi-agent code generation evaluation system according to claim 1, characterized in that: The management agent module includes a knowledge base, a database, a feature engineering and a strategy library; The knowledge base is used to store code generation and evaluation domain knowledge, as well as API documentation and usage examples of various large language models, and is regularly updated to include the latest technology trends and programming paradigms; The database is used for trend analysis and model performance comparison. The database stores multiple benchmark data sets, includes code example libraries for different programming languages ​​and task types, and can maintain historical evaluation data. The feature engineering: according to different test tasks, model performance requirements and previous evaluation result feedback, the raw data is processed and converted to extract features with high correlation or importance with the target variable, and the evaluation data set is optimized and input into the interactive multi-agent module for training; The strategy library is used to store optimization strategies for evaluation indicators, dynamic weight adjustment strategies for evaluation indicators, strategies for data set optimization, strategies for intelligent agent collaboration and task allocation, and strategies for system self-diagnosis and performance tuning.

3. The multi-index scalable multi-agent code generation evaluation system according to claim 1, characterized in that: The interactive multi-agent module includes a data set training module, a large language model interaction module, a data conversion module and an autonomous agent module; The data set training module is used to input multiple benchmark data sets that have been processed by feature engineering in the management agent module and are specifically targeted at code generation for training; The large language model interaction module includes multiple different agents, each of which initializes the API client and configuration according to the assigned large language model and the corresponding API key, communicates with multiple large language models through different API call mechanisms, and generates code results; The data conversion module: performs internal standardization conversion on the data format differences of codes generated by different agents, and inputs the data to the verification agent module; The autonomous agent module: automatically generates more new samples under the original sample training of the benchmark data set according to the feedback of the evaluation results of the management intelligent agent module to form a supplementary data set.

4. The multi-index scalable multi-agent code generation evaluation system according to claim 3, characterized in that: An asynchronous communication protocol based on message passing is used between multiple agents, including: Message format: Use the unified JSON format to encapsulate the message, including the message type, sender, receiver, timestamp and payload data; Message queue: A distributed message queue system is used to manage message transmission between agents, reducing direct dependencies between agents. Subscription mode: Agents can subscribe to messages of corresponding types to achieve flexible many-to-many communication.

5. The multi-index scalable multi-agent code generation evaluation system according to claim 3, characterized in that: A load balancing and fault tolerance mechanism is designed for multi-agents, including: Resource monitoring: real-time monitoring of system resource usage and optimization of task allocation; Elastic expansion: supports dynamic increase or decrease in the number of agents to adapt to load changes; Independent deployment: Supports independent version control and deployment of agents. Updates to a single agent do not affect other agents.

6. The multi-index scalable multi-agent code generation and evaluation system according to claim 1, characterized in that: The benchmark evaluation indicator module adopts a multi-dimensional indicator stacking method: for multiple evaluation indicators such as functional correctness, code quality, code complexity, readability and maintainability, security and efficiency, a dynamic weight calculation method is adopted to adjust the weight of each indicator according to the specific needs and importance of the task, where the dynamic weight calculation formula is: , Where: —weight of indicator i; —normalized score of indicator i; Dynamic weight assignment factors include task type, domain characteristics, user preferences, and historical evaluation data.

7. A multi-index scalable multi-agent code generation evaluation method, characterized by: A multi-index scalable multi-agent code generation evaluation system based on any one of claims 1-6 comprises the following steps: Step 1: The management agent module performs system initialization and task allocation; Step 2: The interactive multi-agent module receives the data set and code generation task processed by the management agent module. Each agent communicates with multiple large language models through different API call mechanisms to generate code results. Step 3: The verification agent module receives the code results in a standardized format from the interactive multi-agent module and performs multi-dimensional evaluation calculations to generate an evaluation report; Step 4: The management agent module receives the evaluation report from the verification agent module, diagnoses, processes and optimizes the evaluation report, and then redistributes it to the interactive multi-agent module to start a new round of evaluation cycle.

8. The multi-index scalable multi-agent code generation evaluation method according to claim 7, characterized in that: The step 1 specifically includes: Step 1.1: The management agent module receives external evaluation task instructions; Step 1.2: Select appropriate benchmark datasets from knowledge bases and databases according to task requirements; Step 1.3: Perform feature engineering on the selected dataset; Step 1.4: Distribute the processed dataset to the interacting multi-agent module.

9. A multi-index scalable multi-agent code generation evaluation method according to claim 8, characterized in that: The feature engineering process is to process and transform the original data according to different test tasks, model performance requirements and previous evaluation result feedback to extract features that are highly correlated with or important to the target variable.

10. The multi-index scalable multi-agent code generation evaluation method according to claim 7, characterized in that: The step 2 specifically includes: Step 2.1: Each agent in the interactive multi-agent module receives the data set processed by the management agent module and the code generation task, including the programming requirements or problem statements described in natural language; Step 2.2: Each agent initializes the API client and configuration according to the assigned large language model and corresponding API key; Step 2.3: The agent converts the task into the format required by the corresponding LLM API; Step 2.4: The agent uses the configured API client to send a request to the corresponding LLM; Step 2.5: The agent receives and parses the generated code returned by LLM; Step 2.6: The agent performs format standardization on the generated code.

11. The multi-index scalable multi-agent code generation evaluation method according to claim 7, characterized in that: The step 3 specifically includes: Step 3.1: The verification agent module receives the generated code in a standardized format from the interactive multi-agent module; Step 3.2: The verification agent evaluates the code in multiple dimensions, including correctness, code quality, complexity, and readability. Step 3.3: The verification agent quantifies and comprehensively calculates the scores of each dimension according to the preset weights; Step 3.4: The verification agent generates a detailed report based on the evaluation results, including the scores of each dimension, the identification of specific problems and suggestions for improvement.

12. A multi-index scalable multi-agent code generation evaluation method according to claim 7, characterized in that: The step 4 specifically includes: Step 4.1: The management agent module receives the evaluation report of the verification agent; Step 4.2: Based on the evaluation results, use the strategy library to diagnose and solve the problem; Step 4.3: Based on the diagnosis results, optimize the data set; Step 4.4: Redistribute the optimized dataset to the interactive multi-agent module and start a new round of evaluation cycle.

Citation Information

Patent Citations

  • Image semantic understanding and text generation based on reinforcement learning and contrast learning

    CN114972907A

  • OCL statement automatic generation method and device based on natural language

    CN115437626A

  • Code processing model training method and device, electronic equipment and storage medium

    CN116820429A

  • Navigation process code generation method based on artificial intelligence

    CN118409742A

  • Multi-agent block chain optimization system and method based on large language model

    CN118657266A