Method for evaluating software intelligent development tool capability facing enterprise-level engineering quality requirements

Abstract

This invention belongs to the field of software engineering technology, specifically relating to a method for evaluating the capabilities of intelligent software development tools to meet enterprise-level engineering quality requirements. The invention includes constructing an evaluation system for intelligent development tools that supports multiple difficulty levels; constructing a multi-type evaluation task set that supports various development scenarios; automating the evaluation of the development assistance capabilities of intelligent development tools; and comprehensively generating multi-dimensional evaluation reports using the data analysis capabilities of a general-purpose large model. This invention supports flexible definition of evaluation task types, difficulty dimensions, and difficulty levels, establishing an evaluation system that is easy to customize and extend. This invention can improve the comprehensiveness and objectivity of intelligent development tool capability evaluation, enabling the evaluation results to objectively reflect the actual assistance effect of intelligent development tools on developers, and providing a reference for evaluating the application effect, product selection, and improvement direction of intelligent development tools for enterprise software development practices.

Description

Technical Field

[0001] This invention belongs to the field of software engineering technology, specifically relating to a method for evaluating the capabilities of intelligent software development tools that meet enterprise-level engineering quality requirements. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, various intelligent development tools based on large models and agents have been widely used in a variety of software development tasks. However, enterprise-level software development has very high requirements for software engineering quality, and the inherent design complexity of the software itself largely limits the actual effectiveness of these intelligent development tools in various development tasks. They often demonstrate excellent development assistance for simple tasks, but when faced with higher design complexity and stricter engineering quality requirements, their final development assistance results are less than satisfactory. How to objectively and comprehensively evaluate the development assistance capabilities of intelligent development tools has become a concern in the industry.

[0003] The academic community has conducted research on the capability evaluation of intelligent development tools based on large models and agents, resulting in numerous benchmark datasets. Datasets such as HumanEval and MBPP primarily evaluate code generation capabilities, requiring the tested intelligent development tools to generate code based on functional descriptions, and then evaluating the generation performance using the similarity between the generated code and pre-defined standard code in a standard dataset. Datasets such as APPS and MultiPL-E also focus on code generation capabilities, but introduce multi-file dependencies, multiple languages, and more context, making the capability evaluation closer to actual engineering development needs. For defect-fixing tasks, SWE-Bench is built based on actual defect information from real open-source projects, focusing on the evaluation of problem understanding, defect localization, and defect fixing capabilities in software maintenance.

[0004] However, existing datasets for evaluating the capabilities of intelligent development tools still suffer from problems such as low design complexity of the target software project itself, overly "clean" problem definitions, limited context, insufficient test completeness leading to misleading evaluation results, and insufficient coverage of software development process capabilities. In various task evaluations, they often fail to fully reflect the actual difficulty levels of problems in complex enterprise-level software systems, cannot handle ambiguous or conflicting requirements, lack common sense and business domain knowledge, ignore coding standards and project conventions, or rely solely on test pass rates to judge task completion. They cannot comprehensively, objectively, and accurately reflect the actual assistance effect of the tested tools on various software development tasks. Therefore, there is a need for a complete measurement index system that meets the needs of evaluating the capabilities of intelligent development tools in complex enterprise-level software systems, covers different development task types and difficulties, reflects the actual effectiveness of assisting development, and includes corresponding automated evaluation methods, supported by a corresponding system, to achieve a systematic and comprehensive evaluation of the capabilities of intelligent development tools. Summary of the Invention

[0005] The purpose of this invention is to provide a method for evaluating the capabilities of intelligent software development tools that meet enterprise-level engineering quality requirements, thereby comprehensively and objectively evaluating the capabilities of various intelligent development tools in completing various auxiliary development tasks of different difficulty levels.

[0006] The intelligent software development tools referred to in this invention are software tools based on large models and agent technology used to assist various software development tasks. This invention is used to evaluate the development assistance capabilities of such tools, and the object being evaluated is referred to as the intelligent development tool under test.

[0007] This invention provides a method for evaluating the capabilities of intelligent software development tools to meet enterprise-level engineering quality requirements. It defines a universal evaluation system for intelligent development tools, provides methods for classifying auxiliary coding tasks and grading task difficulty, and determines the required set of indicators. Based on this, it constructs various types of evaluation task sets for different development scenarios, then conducts capability evaluations of the intelligent development tools. Finally, it automatically generates an evaluation report by integrating the results of various evaluation indicators. The specific steps are as follows: (1) Construct an intelligent development tool evaluation system that supports multiple difficulty levels, which includes four sub-steps: (1.1) Define the evaluation task type; (1.2) Define the difficulty dimensions and levels of the assessment task; (1.3) Define the set of available evaluation indicators; (1.4) Define the final score calculation; (2) Based on the evaluation system constructed in step (1), construct a multi-type evaluation task set that supports multiple development scenarios; specifically, it includes five sub-steps: (2.0) Determine the difficulty dimensions and difficulty level range; (2.1) Obtain the raw data; (2.2) Construct the evaluation task; (2.3) Multi-dimensional difficulty level assessment; (2.4) Calculate the overall difficulty of the evaluation task; (3) Based on the evaluation system constructed in step (1) and the evaluation task set constructed in step (2), the auxiliary development capabilities of intelligent development tools are evaluated in an automated manner; specifically, it includes five sub-steps: (3.1) Create a single evaluation task set; (3.2) Select the indicators and determine the automatic calculation method; (3.3) Perform all evaluation tasks; (3.4) Collect relevant data on the evaluation indicators and calculate the indicator measurement results; (3.5) Calculate the final evaluation results; (4) Based on the evaluation results of step (3), a multi-dimensional evaluation report is generated by utilizing the data analysis capabilities of a general large model; specifically, it includes three sub-steps: (4.1) A comprehensive description of the ability of the tested auxiliary development tool to understand the objectives of a given auxiliary coding task; (4.2) A comprehensive description of the ability of the tested auxiliary development tool to generate the result code or document for a given auxiliary coding task objective; (4.3) Based on the total score and detailed scores of each dimension obtained in sub-step (3.5), automatically provide a summary analysis of the strengths and weaknesses of the auxiliary development capabilities of the tested auxiliary development tool.

[0008] Furthermore: The evaluation task described in step (1) includes direct task input, indirect task input, and expected task output. Direct task input consists of the prompts that constitute the evaluation input for the tested auxiliary development tool, including all directly given code, text, and corresponding requirement instructions. Among them: Indirect inputs to a task include the context within the code repository required to solve the task and the knowledge outside the code repository required to solve the task. The context within the code repository required to solve the task refers to the code or documentation content that is manually defined and needs to be used within the code repository in order to solve a given task. The knowledge outside the code repository required to solve the task refers to the knowledge or data that exists outside the code repository that needs to be used in order to solve a given task.

[0009] The expected task outputs include the code, text, and related instructions that should be generated for each type of evaluation task. It also includes the necessity (i.e., whether the output of the tested auxiliary development tool must contain this content) and compliance (i.e., the degree to which the output of the tested auxiliary development tool conforms to the expected output content) requirements from the perspective of evaluation results.

[0010] Sub-step (1.1) in step (1) defines the evaluation task type, specifically including: (1.1.1) First, it is necessary to determine the input and output formats of the evaluation tasks supported by the evaluation system; among them, the input formats of the evaluation tasks include code, natural language or other forms of input, and the output formats of the evaluation tasks are new code, modifications to the original code, and natural language. (1.1.2) Secondly, for code, it is necessary to determine the types of programming languages ​​supported, such as Java and Python; for natural language, it is necessary to determine the supported languages, such as Chinese and English; for other forms of input and output, it is necessary to determine the media format and format, such as JPEG format images and DOC format documents. (1.1.3) Finally, based on the logical relationship between the input and output content, determine the objectives of the evaluation task, such as code generation and code problem solving.

[0011] Sub-step (1.2) in step (1) defines the difficulty dimension and level of the evaluation task, specifically including: (1.2.1) Determine the input difficulty dimensions and difficulty levels of the evaluation task. Dimensions include heterogeneity, dispersion, scale, domain knowledge, modality, and other custom dimensions. Levels are measured from 0 to N, where N can be set to 3, 5, or 9 as needed to accommodate different levels of difficulty detail. Specifically: (1.2.1.a) Profanity refers to the degree to which the input content, directly or indirectly, contains content unrelated to the task at hand; Level 0: virtually no profanity; Level N: highly profane; Levels 1 to N-1: in between. (1.2.1.b) Dispersion refers to the degree of dispersion of indirect inputs in related code, documents or other knowledge carriers; Level 0: basically not dispersed; Level N: dispersed in many places over a wide range; Level 1 to N-1: in between; (1.2.1.c) Size refers to the length of the input content; Level 0: very short; Level N: very long (e.g., equivalent to more than 30K tokens); Levels 1 to N-1: in between; (1.2.1.d) Domain knowledge refers to the scope of knowledge required to solve a task; Level 0: Only knowledge that can be obtained from public information (including basic programming knowledge and general domain knowledge) is required, and private domain knowledge (knowledge that cannot be directly obtained from public information) is not required; Level N: Private domain knowledge is required to solve a problem; Level 1 to N-1: These are in between, and the lower the level, the less private domain knowledge is required; (1.2.1.e) Modality refers to the degree to which the input contains multiple modalities; multiple modalities refer to text (including code, documentation, bug reports, stack traces, etc.) and images (e.g., screenshots of errors); Level 0: plain text monomodality, Level N: mixed image, document, and text modalities, Level 1 to N-1: in between.

[0012] (1.2.2) Determine the output difficulty dimensions and difficulty levels of the evaluation task. Dimensions include heterogeneity, dispersion, scale, modality, and other custom dimensions. Levels are measured from 0 to N, where N can be set to 3, 5, or 9 to accommodate different levels of detail in difficulty. Specifically: (1.2.2.a) Profanity refers to the proportion of the final output content in all the content of a given physical boundary (such as a function or file) object; Level 0: virtually no profanity, Level N: highly profane, Level 1 to N-1: in between; (1.2.2.b) Scattering refers to the degree to which the final output involves multiple objects with different physical boundaries (such as functions, files); Level 0: basically not scattered, Level N: scattered in many places over a large area, Level 1 to N-1: in between; (1.2.2.c) Scale refers to the length of the final output content; Level 0: very short, Level N: very long, Level 1 to N-1: in between; (1.2.2.d) Modality refers to the required form of the final output content, such as code, document, and graphics; Level 0: plain text monomodality, Level N: mixed image, document, and text multiple modalities, Level 1 to N-1: in between.

[0013] Sub-step (1.3) in step (1) defines the set of available evaluation metrics, specifically including: (1.3.1) Determine the general metrics for the process of the auxiliary development tool under test performing the evaluation task, including thinking time, response time, number of tokens consumed, cost incurred, and other custom metrics; if the auxiliary development tool under test responds in multiple rounds of interaction, record the above data for each round of interaction, as well as the number of rounds. (1.3.2) Determine specific metrics for the output code content, including syntax accuracy, test pass rate, static analysis violations, security issues, performance issues, code content similarity, and other custom metrics; (1.3.3) Determine specific metrics for the output text content, including semantic accuracy, semantic coverage, semantic relevance, factual errors, and other custom metrics; (1.3.4) Determine specific metrics for outputting other modal content, including format correctness, content correctness, content coverage, content relevance, and other custom metrics.

[0014] Step (2) constructs a multi-type evaluation task set that supports various development scenarios, including: (2.0) Determine the difficulty dimension and difficulty level range: Based on steps (1.2.1) and (1.2.2), select the difficulty dimension (the total number of dimensions is M), and determine the specific value of the highest difficulty level N (select one from 3, 5, and 9). (2.1) Obtain raw data: Based on the content and form of the input and output of the software development capability evaluation task determined in steps (1.1.1) and (1.1.2), and the evaluation task objectives determined in step (1.1.3), obtain the raw data of the evaluation task from open source or private code repositories, including the code required for the software-specific software development task, issue forms (i.e. defect reports), code submissions, etc. (2.2) Construct the evaluation task: Based on the data obtained in step (2.1) and the basic content requirements of the evaluation task in step (1), identify the direct input, indirect input and expected task output of the evaluation task; the identification method can be automatic or manual. (2.3) Multi-dimensional difficulty rating: Based on the difficulty dimensions and levels in steps (1.2.1) and (1.2.2), for the direct inputs, indirect inputs and expected outputs identified in step (2.2), the difficulty level of the difficulty dimension is determined. The difficulty level is an integer ranging from 0 to N selected in sub-step (2.0) (inclusive of 0 and the selected N). (2.4) Calculate the overall difficulty of the assessment task; the final overall difficulty is obtained through the following two methods: (2.4.a) Normalized Residual Method for Highest Difficulty: The final difficulty level of the evaluation task is determined by superimposing the highest difficulty level of each dimension with the difficulty of other dimensions; the specific method is as follows: ,in x i Difficulty value for each dimension, x maxThe maximum value among all the difficulty values ​​set in sub-step (2.3), where D is the overall difficulty level, M is the total number of difficulty dimensions selected in sub-step (2.0), and N is the highest difficulty level value set in sub-step (2.0); for ease of use, D Accurate to 1; (2.4.b) Multidimensional radar chart method; The method of determining the final difficulty level of the evaluation task using the multidimensional radar chart method is as follows: retain the original difficulty level of each dimension, take each dimension of input and output as the radiation axis, and construct a radar chart with a scale of 0 to N levels (step size of 1) on the axis; take the final form of the radar chart as the final difficulty form of the evaluation task, and determine the comprehensive difficulty level based on the area of ​​the graphic enclosed by the radar chart.

[0015] Based on the above calculation method, the final overall difficulty level of the evaluation task is from 0 to N (accurate to 0.1), and the specific difficulty level of each difficulty dimension can be viewed.

[0016] Step (3) involves evaluating the assisted development capabilities of the intelligent development tools in an automated manner, wherein: Sub-step (3.1) Create a single evaluation task set: Specifically, from the multi-type evaluation task set constructed in step (2), select evaluation tasks that meet the requirements and cover different difficulty levels as all the evaluation tasks required for a single evaluation of the intelligent development tool under test; randomly select n tasks (n≥5) from the evaluation tasks corresponding to each type of development scenario, in the range of comprehensive difficulty level 0 to N, in the high, medium and low segments; for example, if N is 9, ensure that the selected tasks cover the difficulty level range of [0,3], (3,6], and (6-N] in each dimension; then, according to the calling method given by the auxiliary development tool under test, take the requirements of each evaluation task as the input of the auxiliary development tool under test, so that the auxiliary development tool under test can complete the required auxiliary development tasks; Sub-step (3.2) Selecting indicators and determining automatic calculation: Specifically, based on sub-steps (1.3.1), (1.3.2), (1.3.3), and (1.3.4), select appropriate evaluation indicators and determine specific automatic calculation methods for each indicator; automatic calculation methods include directly recording values ​​(such as running time), formulating calculation formulas (such as test pass rate), and judging based on the large-scale judge model (such as semantic accuracy and semantic coverage). Sub-step (3.3) executes all evaluation tasks: Specifically, for each evaluation task in the evaluation task set determined in sub-step (3.1), based on the calling method given by the auxiliary development tool under test, the direct input of each evaluation task is used as the input of the auxiliary development tool under test, and necessary external data such as code repositories are provided to the auxiliary development tool under test, so that the auxiliary development tool under test can complete the corresponding auxiliary development tasks as required. Sub-step (3.4) collects relevant data on evaluation indicators and calculates indicator measurement results: Specifically, it collects the evaluation task execution process and result data from sub-step (3.3), and then automatically calculates and records the quantitative calculation results of each indicator based on the indicators and their automatic calculation methods determined in sub-step (3.2). Sub-step (3.5) calculates the final evaluation result: Specifically, based on the calculation results of each indicator in sub-step (3.4), the final comprehensive score is calculated according to the following steps: (3.5.1) Score Calculation for Individual Indicators: For the indicators determined in sub-step (3.2), for directly recorded numerical indicators, a benchmark value is selected, and the percentage deviation of the actual measured value from the benchmark value is multiplied by 100 to record the deviation adjustment value as a percentage score; for indicators that need to be calculated, if the calculation result is a percentage, the percentage is directly multiplied by 100 to convert it into a percentage score; if the calculation result is a numerical value, a benchmark value is selected, and the percentage deviation of the actual measured value from the benchmark value is calculated, and the percentage is multiplied by 100 to record it as a percentage score; for indicators obtained from the judgment of the large-scale judge model, they are converted into percentage scores according to the judgment result; the score of indicator i is recorded as s i ; (3.5.2) Score calculation for a single evaluation task: Based on the calculation results of each indicator in sub-step (3.5.1), determine the weight of each indicator i according to the type of the evaluation task. w i And calculate the weighted sum to obtain the percentage score for a single evaluation task t. ; (3.5.3) Calculation of the difficulty-weighted comprehensive score of all evaluation tasks: based on the evaluation task t of sub-step (3.5.2). i Percentage score S ti Multiply by the overall difficulty of each evaluation task determined based on sub-step (2.4.a). D ti And sum them up to get the absolute score. Normalize the absolute scores to a percentage system to obtain the percentage score. .

[0017] Step (4) involves generating a multi-dimensional evaluation report. The specific steps are as follows: (4.1) A comprehensive description of the ability of the tested auxiliary development tool to understand the given auxiliary coding task objectives, including the accuracy and completeness of understanding direct input and the completeness and accuracy of acquiring indirect input; (4.2) A comprehensive description of the ability of the tested auxiliary development tool to generate the result code or document of a given auxiliary coding task objective, including the accuracy and completeness of the generated content, as well as the degree of completion of the explicit or implicit requirements in the task instructions; (4.3) Based on the total score and detailed scores of each dimension obtained in sub-step (3.5), automatically provide a summary analysis of the strengths and weaknesses of the auxiliary development capabilities of the tested auxiliary development tool.

[0018] The present invention also includes a system constructed for implementing the above method, specifically including: (S1) an evaluation task dataset construction subsystem; and (S2) an automated evaluation subsystem.

[0019] Furthermore: In (S1) above, various evaluation tasks are first collected from open source projects. The collection and construction methods may vary depending on the type of evaluation task, but they are all ultimately represented as the direct and indirect inputs, expected outputs, and specific metrics and measurement methods for the expected outputs required by the evaluation task. Then, the task difficulty automatic classification method is used to determine the difficulty level of the task in each dimension. Finally, the standardized evaluation task data is put into the evaluation task dataset for use by the automated evaluation subsystem.

[0020] In step (S2) above, the access method (such as API Key) of the auxiliary development tool under test is first obtained; then, for each evaluation task type, a certain number of evaluation tasks are extracted from three difficulty levels (easy, medium, and hard) according to a certain proportion, and the direct input of the evaluation tasks is sent to the auxiliary development tool under test through the access method of the auxiliary development tool under test; next, the evaluation results and analysis results are obtained according to the automatic evaluation method steps; finally, the evaluation results and analysis results are output to the final evaluation report.

[0021] The final evaluation report (S2) above includes the actual output results of each type of evaluation task, the measurement results of each dimension, and the evaluation conclusions.

[0022] Compared with the prior art, the present invention has the following advantages and positive effects: This invention supports flexible definition of evaluation task types, difficulty dimensions and levels, and available indicator sets, establishing an evaluation system that is easy to customize and expand. When constructing the evaluation task dataset, it fully considers the design complexity of enterprise-level software, providing specific evaluation methods from aspects such as evaluation task type setting, task difficulty grading dimensions, and measurement indicator settings. It automatically executes evaluation tasks and provides comprehensive evaluation results. This invention will help improve the comprehensiveness and objectivity of intelligent development tool capability evaluation, enabling evaluation results to objectively reflect the actual assistance effect of intelligent development tools on developers, and providing a reference for evaluating the application effect of intelligent development tools in enterprise software development practices, product selection, and improvement directions. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the basic process of the present invention. It shows five steps: constructing an evaluation task, determining the difficulty of the evaluation task, selecting and executing the evaluation task, measuring the task execution process and results, and generating a comprehensive evaluation result report.

[0024] Figure 2 The diagram illustrates a typical implementation process of the present invention. First, it shows the basic process of constructing a specific multi-dimensional difficulty level intelligent development tool evaluation system. Then, taking a "problem location and repair" task in an open-source project as an example, it shows the construction and difficulty classification of the evaluation task of "problem location and solution". It also assumes that after the evaluation task dataset is added, the specific implementation process of evaluating the auxiliary development capabilities of three different intelligent development tools is carried out. Detailed Implementation

[0025] The present invention will be further described below with reference to the embodiments and accompanying drawings.

[0026] Figure 2 Taking an issue from an open-source project as an example, this paper demonstrates how to define the task type (1.1), difficulty dimensions and levels (1.2), and available indicator sets (1.3) for the "problem identification and resolution" evaluation task. Since the purpose of step (1) is to build a complete evaluation system, therefore, in Figure 2 The text only schematically shows the content, difficulty dimensions and levels, and indicators of the "problem location and solution" type of assessment task after completing step (1).

[0027] According to step (1.1.1), the input for the "problem location and resolution" type of problem is a problem description expressed in natural language (direct input) and the corresponding code repository (indirect input of the search scope).

[0028] According to step (1.1.2), for the evaluation system constructed in this study, the programming language for task input only supports TypeScript, and the natural language only supports English; no other media forms or formats are involved.

[0029] According to step (1.1.3), the input consists of a problem description and related code repositories. The output should be the location and repair of defects in the code based on an understanding of the problem description. Therefore, the evaluation task of "Problem Location and Resolution" is determined. This type only supports TypeScript and is described in English.

[0030] It should be noted that if you want to build an evaluation task set that supports multiple languages, simply set the desired supported languages ​​in the steps above. This will not be elaborated further in this example.

[0031] Based on step (1.2.1), this type of evaluation task requires five input difficulty dimensions: mixed cases, dispersion, scale, domain knowledge, and modality. No custom input difficulty dimensions are needed. Based on step (1.2.2), four output difficulty dimensions are needed: mixed cases, dispersion, scale, and modality. No custom output difficulty dimensions are needed. There are a total of nine difficulty dimensions.

[0032] Based on step (1.3.1), the general metrics required for this type of evaluation task are: thinking time, response time, number of tokens consumed, and cost incurred. Multi-round interactions are not supported. This will result in an evaluation system that supports single-round interactions.

[0033] Based on step (1.3.2), the specific metrics required for the output code of this type of evaluation task are determined as follows: syntax accuracy, test pass rate, static analysis of violations, and code content similarity.

[0034] Based on steps (1.3.3) and (1.3.4), it is determined that this type of evaluation task does not require the output of text content or other modal content, and therefore no related indicators are needed.

[0035] This approach allows you to define other assessment task types, as well as the required difficulty dimensions, levels, and available metrics. Based on this, you can then begin building your assessment task set.

[0036] Based on step (2.0), and building upon previous steps (1.2.1) and (1.2.2), the total number of difficulty dimensions is determined to be M=9, and the highest difficulty level is N=9. This step essentially confirms the result of step (1). Figure 2 It is not shown separately in the document.

[0037] Let's take the construction of the "problem identification and resolution" evaluation task and the process of using this task to automatically evaluate the development capabilities of large models as an example.

[0038] First, according to step (2.1), analyze the Issue list in the open source code repository where the provided open source project is located, obtain the description of an Issue, and the actual code modification content to fix the Issue. Figure 2 The left half provides an illustration.

[0039] Then, construct an evaluation task according to step (2.2). The direct input of this task includes a description of the Issue, a description of the basic information of the project containing the Issue, and an instruction description of the development aid under test to solve the problem. The indirect input required for this task is the code files and functions modified when actually fixing the problem. The expected output of this task is the code modifications that actually fix the problem.

[0040] Based on step (2.3), determine the difficulty of the task. First, consider the input. According to step (1.2.1.a), the given input includes a description of the issue, with relatively low clutter; however, it also includes the entire code repository of the open-source project, containing a large amount of code unrelated to solving the issue. In particular, the functions containing the issue-related code contain other unrelated content, resulting in high clutter. Overall, the clutter level is 8. According to step (1.2.1.b), the indirect input required for this task is scattered across four different files, involving three different folders, and it is difficult to obtain the correct location directly through keyword search. Therefore, the dispersion level is high, rated as 7. According to step (1.2.1.c), the direct input is relatively short, but since the indirect input may involve multiple parts of the entire code repository, the level is rated as 5. According to step (1.2.1.d), solving this problem requires a thorough understanding of the concepts described, such as "members," "groups," and "alarm information," and their interrelationships. This understanding needs to be derived from the implementation of the code repository and is not entirely based on programming common sense and general knowledge. Therefore, it has a certain level of domain knowledge difficulty and is rated as 6. According to step (1.2.1.e), the input content appears only in code and text form and is not complex, so it is rated as 2.

[0041] Next, consider the output. According to steps (1.2.2.a) and (1.2.2.b), fixing the issue requires code modifications in four different files across three different directories. Furthermore, the content of each file and function is not entirely directly related to the issue; therefore, the messiness and dispersion are both moderately high, resulting in a rating of 6. According to step (1.2.2.c), the actual code output and the repair code are not long, and no further explanatory text is required; therefore, the output scale dimension is low, rated at 3. Finally, according to step (1.2.2.d), the output only includes the location and content of the repair code, which must meet general code requirements and is relatively simple; therefore, it is rated at 2.

[0042] Calculate the overall difficulty of the assessment task according to step (2.4). Using the highest difficulty normalized residual method (2.4.a), the final difficulty is 8 + (9-8)*[(7+5+6+2+6+6+3+3) / 72] 2 ≈8.3; Using the multidimensional radar chart method (2.4.b), a radar chart with 9 axes will be formed, with points selected within the range of 0-9 on each axis according to the difficulty level. In practical use, if the overall difficulty of the task is selected, the method in step (2.4.a) is usually used; if each individual dimension is viewed and selected, the method in step (2.4.b) is usually used.

[0043] Using a similar approach, difficulty levels can be determined for other types of evaluation tasks, forming an evaluation task dataset.

[0044] Then, according to step (3), select the appropriate number of evaluation tasks. The automatic evaluation system will use the evaluation tasks as input to the auxiliary development tool under test, and then obtain the execution process and output results of the auxiliary development tool under test. Finally, calculate the final evaluation result score for each task. In this example, assume that 15 evaluation tasks were selected through step (3.1), and all of them are of the "problem location and solution" type. Figure 2 The specific problem shown is considered a difficult task (difficulty level 8.3) among the 15 evaluation tasks; assuming the difficulties of the other 14 tasks are 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5 respectively, it meets the difficulty distribution requirements.

[0045] Following step (3.2), the following metrics are selected: coverage of indirect inputs (I1), compilation success rate (I2), test pass rate (I3), similarity between modified and expected modifications (I4), and static analysis of violations (I5). It is agreed that the similarity between modified and expected modifications will only be used when compilation or testing fails. To demonstrate the differences in evaluation results among different tested intelligent development tools, for the above tasks, we consider the implementation of three different tested auxiliary development tools after performing the evaluation task according to step (3.3). The steps are: obtaining relevant metric measurement results in step (3.4), calculating individual metric scores in step (3.5.1), and calculating individual evaluation task scores in step (3.5.2). Without affecting generality, the general metric data defined in (1.3.1) are not shown in this example.

[0046] The tested auxiliary development tool A successfully located the files and functions that needed modification, achieving 100% coverage in indicator (I1). Since this is percentage data, it receives 100 points. It also provided correct repair results, passing compilation and corresponding tests; therefore, indicator (I2) is "passed," and since the benchmark value is "passed," it receives 100 points. Indicator (I3) also has a 100% pass rate, resulting in a score of 100. No further evaluation is needed for indicator (I4). Finally, indicator (I5) is checked, revealing one significant static analysis violation in the output code. Since the benchmark value for (I5) is set to 0 (i.e., no significant static analysis violation), and 30 points are deducted for each violation, indicator (I5) receives 70 points. According to (3.5.2), assuming each indicator weight is set to 1, the final comprehensive score is (100+100+100+70) / 4 = 92.5 points.

[0047] The tested auxiliary development tool B only successfully located the files and functions that needed modification, but failed to locate some content. Therefore, based on the location, the code line-level accuracy and coverage were calculated, resulting in indicator (I1) with 80% coverage and a score of 80. Although the final repair code failed to compile, and the test cases could not be executed, resulting in failure of evaluations (I2) and (I3), the scores were 0 respectively. According to the rules, the repair code content needs to be reviewed, and indicator (I4) needs to be checked. From the actual modified code, the key logic and key function calls are basically correct, with errors only occurring in local areas such as function parameter passing, causing compilation errors. The consistency between the output code and the expected code reached 95%. After fixing the compilation problem, it can pass the test, so the score for I4 is 95. In addition, the output code, like tool A, also has one important static analysis violation problem, so (I5) gets 70 points. According to step (3.5.2), assuming that the weight of each indicator is set to 1, the final percentage score is (80+0+0+95+70) / 5=49 points.

[0048] The tested auxiliary development tool C failed to locate the key files and functions. Although it eventually compiled, all test cases failed. Furthermore, the output code content differed from the expected output. Therefore, the model received a score of 0 on this task.

[0049] In step (3.5.3), the difficulty of this example assessment task has been set to 8.3 according to step (2.4.a). The weighted score for tool A is 92.5 × 8.3 = 767.75, for B it is 406.7, and for C it is 0. Since 15 tasks were selected according to (3.1), the overall score can be calculated based on the comprehensive score of each tool on the specific task and (3.5.3).

[0050] Finally, in step (4), the corresponding evaluation results are output for the three tested auxiliary development tools. In the example issue location and repair task, the tested auxiliary development tool A has relatively comprehensive capabilities, the tested auxiliary development tool B has shortcomings in the accuracy of code output, resulting in uncompileability, but its overall capabilities are strong and it can still play a good role in engineering practice; the tested auxiliary development tool C has poor overall capabilities.

[0051] The final evaluation results obtained based on this invention demonstrate that it does not solely rely on a single metric to judge the development-aiding capabilities of large-model systems. For example, even if compilation or testing fails, the evaluation can still assess the completion of the assessment task by considering factors such as the coverage and accuracy of indirect inputs, or the actual feedback provided to developers by the generated content. This provides a more comprehensive evaluation of the practical role of large-model-based development-aiding tools in software development practices with enterprise-level engineering quality requirements, and is more aligned with developers' actual experience.

Claims

1. A method for evaluating the capabilities of intelligent software development tools oriented towards enterprise-level engineering quality requirements, characterized in that, By defining a general evaluation system for intelligent development tools, this paper presents a classification method for auxiliary coding tasks, a task difficulty grading method, and determines the required set of indicators. Based on this, various types of evaluation task sets are constructed for different development scenarios, and then the capabilities of intelligent development tools are evaluated. Finally, an evaluation report is automatically generated by integrating the results of various evaluation indicators. The specific steps are as follows: (1) Construct an intelligent development tool evaluation system that supports multiple difficulty levels, which includes four sub-steps: (1.1) Define the evaluation task type; (1.2) Define the difficulty dimensions and levels of the assessment task; (1.3) Define the set of available evaluation indicators; (1.4) Define the final score calculation; (2) Based on the evaluation system constructed in step (1), construct a multi-type evaluation task set that supports multiple development scenarios; specifically, it includes five sub-steps: (2.0) Determine the difficulty dimensions and difficulty level range; (2.1) Obtain the raw data; (2.2) Construct the evaluation task; (2.3) Multi-dimensional difficulty level assessment; (2.4) Calculate the overall difficulty of the evaluation task; (3) Based on the evaluation system constructed in step (1) and the evaluation task set constructed in step (2), the auxiliary development capabilities of intelligent development tools are evaluated in an automated manner; specifically, it includes five sub-steps: (3.1) Create a single evaluation task set; (3.2) Select the indicators and determine the automatic calculation method; (3.3) Perform all evaluation tasks; (3.4) Collect relevant data on the evaluation indicators and calculate the indicator measurement results; (3.5) Calculate the final evaluation results; (4) Based on the evaluation results of step (3), a multi-dimensional evaluation report is generated by utilizing the data analysis capabilities of a general large model; specifically, it includes three sub-steps: (4.1) A comprehensive description of the ability of the tested auxiliary development tool to understand the objectives of a given auxiliary coding task; (4.2) A comprehensive description of the ability of the tested auxiliary development tool to generate the result code or document for a given auxiliary coding task objective; (4.3) Based on the total score and detailed scores of each dimension obtained in sub-step (3.5), automatically provide a summary analysis of the strengths and weaknesses of the auxiliary development capabilities of the tested auxiliary development tool.

2. The method for evaluating the capabilities of intelligent software development tools according to claim 1, characterized in that, The evaluation task described in step (1) includes direct task input, indirect task input, and expected task output; direct task input consists of prompts that constitute the evaluation input of the development aid under test, including all directly given code, text, and corresponding requirements; wherein: Indirect inputs to a task include the context within the code repository required to solve the task and the knowledge outside the code repository required to solve the task. The context within the code repository required to solve the task refers to the code or documentation content that is manually defined and needs to be used within the code repository in order to solve a given task. The knowledge outside the code repository required to solve the task refers to the knowledge or data that exists outside the code repository that needs to be used in order to solve a given task. The expected task outputs include the code, text, and related instructions that should be generated for each type of evaluation task. From the perspective of evaluation results, it also includes the necessity of each type of output content, i.e., whether the output content of the tested auxiliary development tool must contain this content, and the compliance, i.e., the degree to which the output content of the tested auxiliary development tool conforms to the expected output content.

3. The method for evaluating the capabilities of intelligent software development tools according to claim 2, characterized in that, Sub-step (1.1) in step (1) defines the evaluation task type, specifically including: (1.1.1) The input and output formats of the evaluation tasks supported by the evaluation system should be determined; the input formats of the evaluation tasks include code, natural language or other forms of input, and the output formats of the evaluation tasks are new code, modifications to the original code, and natural language. (1.1.2) For code, determine the types of programming languages ​​supported; for natural language, determine the languages ​​supported; for other forms of input and output, determine the media format and format. (1.1.3) Determine the objectives of the evaluation task based on the logical relationship between the input and output content, including code generation and code problem solving.

4. The method for evaluating the capabilities of intelligent software development tools according to claim 3, characterized in that, Sub-step (1.2) in step (1) defines the difficulty dimension and level of the evaluation task, specifically including: (1.2.1) Determine the input difficulty dimensions and difficulty levels of the evaluation task. Dimensions include heterogeneity, dispersion, size, domain knowledge, modality, and other custom dimensions. Levels are measured from 0 to N, where N is set to one of 3, 5, or 9 based on actual needs to accommodate different levels of difficulty detail. Specifically: (1.2.1.a) Profanity refers to the degree to which the input content, directly or indirectly, contains content unrelated to the task at hand; Level 0: virtually no profanity; Level N: highly profane; Levels 1 to N-1: in between. (1.2.1.b) Dispersion refers to the degree of dispersion of indirect inputs in related code, documents or other knowledge carriers; Level 0: basically not dispersed; Level N: dispersed in many places over a wide range; Level 1 to N-1: in between; (1.2.1.c) Size refers to the length of the input content; Level 0: very short; Level N: very long; Levels 1 to N-1: in between; (1.2.1.d) Domain knowledge refers to the scope of knowledge required to solve a task; Level 0: Knowledge that can only be obtained from publicly available information, including basic programming knowledge and general domain knowledge, without the need for private domain knowledge, and knowledge that cannot be directly obtained from publicly available information; Level N: Private domain knowledge is required to solve a problem; Level 1 to N-1: Between the two, and the lower the level, the less private domain knowledge is required; (1.2.1.e) Modality refers to the degree to which the input contains multiple modalities; multiple modalities refer to text and images; Level 0: plain text monomodality, Level N: mixed images, documents, and text multiple modalities; Level 1 to N-1: in between; (1.2.2) Determine the output difficulty dimensions and difficulty levels of the evaluation task. Dimensions include heterogeneity, dispersion, scale, modality, and other custom dimensions. Levels are measured from 0 to N, where N is set according to actual needs to accommodate different levels of detail in difficulty. Specifically: (1.2.2.a) Profanity refers to the proportion of the final output content in all content of a given physical boundary object; Level 0: virtually no profanity, Level N: highly profane, Level 1 to N-1: in between; (1.2.2.b) Scattering refers to the degree to which the final output involves multiple objects with different physical boundaries; Level 0: basically not scattered, Level N: scattered in many places over a large area, Level 1 to N-1: in between; (1.2.2.c) Scale refers to the length of the final output content; Level 0: very short, Level N: very long, Level 1 to N-1: in between; (1.2.2.d) Modality refers to the required form of the final output content, including code, document, and graphics; Level 0: plain text monomodality; Level N: mixed image, document, and text modality; Level 1 to N-1: in between.

5. The method for evaluating the capabilities of intelligent software development tools according to claim 4, characterized in that, Sub-step (1.3) in step (1) defines the set of available evaluation metrics, specifically including: (1.3.1) Determine the general metrics for the process of the auxiliary development tool under test performing the evaluation task, including thinking time, response time, number of tokens consumed, cost incurred, and other custom metrics; if the auxiliary development tool under test responds in multiple rounds of interaction, record the above data for each round of interaction, as well as the number of rounds. (1.3.2) Determine specific metrics for the output code content, including syntax accuracy, test pass rate, static analysis violations, security issues, performance issues, code content similarity, and other custom metrics; (1.3.3) Determine specific metrics for the output text content, including semantic accuracy, semantic coverage, semantic relevance, factual errors, and other custom metrics; (1.3.4) Determine specific metrics for outputting other modal content, including format correctness, content correctness, content coverage, content relevance, and other custom metrics.

6. The method for evaluating the capabilities of intelligent software development tools according to claim 3, characterized in that, Step (2) Construct a multi-type evaluation task set that supports multiple development scenarios, including: Sub-step (2.0) determines the difficulty dimension and difficulty level range: based on steps (1.2.1) and (1.2.2), select the difficulty dimension, the total number of dimensions is M, and determine the specific value of the highest difficulty level N; Sub-step (2.1) Obtaining raw data: Based on the content and form of the input and output of the software development capability evaluation task determined in steps (1.1.1) and (1.1.2), and the evaluation task objectives determined in step (1.1.3), the raw data of the evaluation task is obtained from open source or private code repositories, including the code, issue forms, and code submissions required for the specific software development task. Sub-step (2.2) constructs the evaluation task: based on the data obtained in step (2.1) and the basic content requirements of the evaluation task in step (1), the direct input, indirect input and expected task output of the evaluation task are identified; the identification method is automatic or manual. Sub-step (2.3) Multi-dimensional difficulty rating: Based on the difficulty dimensions and levels of steps (1.2.1) and (1.2.2), for the direct inputs, indirect inputs and expected outputs identified in step (2.2), determine the difficulty level of the difficulty dimension. The difficulty level is an integer between 0 and N selected in sub-step (2.0). Sub-step (2.4) Calculate the overall difficulty of the evaluation task: The final overall difficulty is obtained through the following two methods: (2.4.a) Normalized Residual Method for Highest Difficulty: The final difficulty level of the evaluation task is determined by superimposing the highest difficulty level of each dimension with the difficulty of other dimensions. The specific method is as follows: ,in x i Difficulty value for each dimension, x max The maximum value among all the difficulty values ​​set in sub-step (2.3), where D is the overall difficulty level, M is the total number of difficulty dimensions selected in sub-step (2.0), and N is the highest difficulty level value set in sub-step (2.0). (2.4.b) Multidimensional radar chart method; The method of determining the final difficulty level of the evaluation task using the multidimensional radar chart method is as follows: retain the original difficulty level of each dimension, take each dimension of input and output as the radial axis, and use the 0 to N level as the scale on the axis with a step size of 1 to construct a radar chart; take the final form of the radar chart as the final difficulty form of the evaluation task, and determine the comprehensive difficulty level based on the area of ​​the graphic enclosed by the radar chart. Based on the above calculation method, the final overall difficulty level of the evaluation task is from 0 to N, and the specific difficulty level of each difficulty dimension can be viewed.

7. The method for evaluating the capabilities of intelligent software development tools according to claim 6, characterized in that, The automated method described in step (3) evaluates the development-aiding capabilities of the intelligent development tool, wherein: Sub-step (3.1) Create a single evaluation task set: Specifically, from the multi-type evaluation task set constructed in step (2), select evaluation tasks that meet the requirements and cover different difficulty levels as all the evaluation tasks required for one evaluation of the intelligent development tool under test; randomly select n tasks from the evaluation tasks corresponding to each type of development scenario, with a comprehensive difficulty level of 0 to N, and three levels of high, medium and low, n≥5; then, according to the calling method given by the auxiliary development tool under test, use the requirements of each evaluation task as the input of the auxiliary development tool under test, so that the auxiliary development tool under test can complete the required auxiliary development tasks; Sub-step (3.2) Selecting indicators and determining automatic calculation: Specifically, based on sub-steps (1.3.1), (1.3.2), (1.3.3), and (1.3.4), select appropriate evaluation indicators and determine specific automatic calculation methods for each indicator; automatic calculation methods include directly recording values, formulating calculation formulas, and judgment based on the large-scale referee model; Sub-step (3.3) executes all evaluation tasks: Specifically, for each evaluation task in the evaluation task set determined in sub-step (3.1), based on the calling method given by the auxiliary development tool under test, the direct input of each evaluation task is used as the input of the auxiliary development tool under test, and necessary external data such as code repositories are provided to the auxiliary development tool under test, so that the auxiliary development tool under test can complete the corresponding auxiliary development tasks as required. Sub-step (3.4) collects relevant data on evaluation indicators and calculates indicator measurement results: Specifically, it collects the evaluation task execution process and result data from sub-step (3.3), and then automatically calculates and records the quantitative calculation results of each indicator based on the indicators and their automatic calculation methods determined in sub-step (3.2). Sub-step (3.5) calculates the final evaluation result: Specifically, based on the calculation results of each indicator in sub-step (3.4), the final comprehensive score is calculated according to the following steps: (3.5.1) Score Calculation for Individual Indicators: For the indicators determined in sub-step (3.2), for directly recorded numerical indicators, a benchmark value is selected, and the percentage deviation of the actual measured value from the benchmark value is multiplied by 100 to record the deviation adjustment value as a percentage score; for indicators that need to be calculated, if the calculation result is a percentage, the percentage is directly multiplied by 100 to convert it into a percentage score; if the calculation result is a numerical value, a benchmark value is selected, and the percentage deviation of the actual measured value from the benchmark value is calculated, and the percentage is multiplied by 100 to record it as a percentage score; for indicators obtained from the judgment of the large-scale judge model, they are converted into percentage scores according to the judgment result; the score of indicator i is recorded as s i ; (3.5.2) Score calculation for a single evaluation task: Based on the calculation results of each indicator in sub-step (3.5.1), determine the weight of each indicator i according to the type of the evaluation task. w i And calculate the weighted sum to obtain the percentage score for a single evaluation task t. ; (3.5.3) Calculation of the difficulty-weighted comprehensive score of all evaluation tasks: based on the evaluation task t of sub-step (3.5.2). i Percentage score S ti Multiply by the overall difficulty of each evaluation task determined based on sub-step (2.4.a). D ti And sum them up to get the absolute score. Normalize the absolute scores to a percentage system to obtain the percentage score. .

8. The method for evaluating the capabilities of intelligent software development tools according to claim 7, characterized in that, Step (4) generates a comprehensive multi-dimensional evaluation report, in which: Sub-step (4.1) provides a comprehensive description of the ability of the tested auxiliary development tool to understand the given auxiliary coding task objective, including the accuracy and completeness of understanding direct input and the completeness and accuracy of acquiring indirect input; Sub-step (4.2) provides a comprehensive description of the ability of the tested auxiliary development tool to generate the target result code or document for a given auxiliary coding task, including the accuracy and completeness of the generated content, as well as the degree of completion of the explicit or implicit requirements in the task instructions; Sub-step (4.3) automatically provides a summary analysis of the strengths and weaknesses of the tested auxiliary development tool's auxiliary development capabilities based on the total score and detailed scores of each dimension obtained in sub-step (3.5).

9. A software intelligent development tool capability evaluation system based on the software intelligent development tool capability evaluation method according to any one of claims 1-8, characterized in that, Specifically, it includes: Evaluation task dataset construction subsystem (S1); automated evaluation subsystem (S2). In S1, various evaluation tasks are first collected from open-source projects. The collection and construction methods may vary depending on the type of evaluation task, but they are all ultimately represented as the direct and indirect inputs required for the evaluation task, the expected output, and the specific metrics and measurement methods for the expected output. Then, the task difficulty automatic classification method is used to determine the difficulty level of the task in each dimension. Finally, the standardized evaluation task data is put into the evaluation task dataset for use by the automated evaluation subsystem. In S2, the access method of the auxiliary development tool under test is first obtained; then, for each evaluation task type, a certain number of evaluation tasks are selected from three difficulty levels (easy, medium, and hard) according to a certain proportion, and the direct input of the evaluation tasks is sent to the auxiliary development tool under test through the access method of the auxiliary development tool under test; next, the evaluation results and analysis results are obtained according to the automatic evaluation method steps; finally, the evaluation results and analysis results are output to the final evaluation report. The final evaluation report includes the actual output results of each type of evaluation task, the measurement results of each dimension, and the evaluation conclusions.

Images