Statement level code abstract generation and automatic evaluation method and system based on large language model

Abstract

The invention discloses a statement level code abstract generation and automatic evaluation method and system based on a large language model, and the method comprises the steps: carrying out the data cleaning and semantic correlation screening of a code and an annotation thereof, removing a low-quality sample, constructing a high-quality data set, and obtaining a statement level code abstract; on this basis, a high-quality statement-level code abstract is generated through a large language model, it is ensured that the generated abstract is highly consistent with code semantics by adopting zero-sample, few-sample, thinking chain and reflection type prompt strategies in combination with code context information, and a comprehensive evaluation score is generated in combination with large language model evaluation and index evaluation; according to the method, manual intervention is reduced, so that the accuracy, context consistency and semantic integrity of code abstract generation are improved, and the objectivity and reliability of evaluation are also improved.

Description

Technical Field

[0001] The present invention relates to a method and system for generating and automatically evaluating a code summary, and in particular to a method and system for generating and automatically evaluating a statement-level code summary based on a large language model. Background Art

[0002] With the rapid development of software engineering, the complexity and maintenance cost of code are gradually increasing. Traditional code annotation methods have many problems, such as the mismatch between annotations and actual code, incomplete or outdated annotations, which weaken the maintainability of software to a certain extent. Based on the above background, automatic code summarization technology has gradually become an effective means to improve code readability and maintainability. In recent years, deep learning models have shown excellent results in code summarization tasks. Deep learning models can extract key information based on learning code structure and logic and generate concise code summaries.

[0003] Although a lot of research has focused on function-level code summarization, there is less research on more fine-grained statement-level code summarization. Statement-level code summarization requires a deep understanding of the contextual information of the statement, which places higher demands on traditional deep learning models. Existing code summary datasets contain a large amount of noisy data, such as comments that are irrelevant or low-relevant to the code function. These noises will seriously affect the performance of automatic code summarization models. In addition, current evaluation methods also have certain limitations, making it difficult to comprehensively and accurately evaluate model performance. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a statement-level code summary generation and automatic evaluation method based on a large language model to reduce data set noise, improve the accuracy and semantic completeness of code summary generation and evaluation efficiency, and on the other hand, to provide a statement-level code summary generation and automatic evaluation system based on a large language model.

[0005] Technical solution: The present invention provides a method for generating and automatically evaluating sentence-level code summaries based on a large language model, comprising the following steps:

[0006] (1) Constructing a code summary dataset: extracting codes and their corresponding comments from code snippets, constructing code-comment pairs, and performing data cleaning and screening. All code-comment pairs constitute a code summary dataset;

[0007] (2) Generate code summaries: Use a large language model, combined with prompt strategies and context enhancement techniques, to generate sentence-level code summaries from the codes in the code summary dataset;

[0008] (3) Evaluate code summaries: Combine the large language model evaluation and indicator evaluation results to generate a comprehensive evaluation score for the code summary.

[0009] Preferably, the data cleaning in step 1 includes deleting all multi-line comments, splitting the code line by line and identifying single-line comments.

[0010] Preferably, in the data cleaning described in step 1, entries containing import or package keywords are removed from non-functional codes.

[0011] Preferably, the data cleaning in step 1 utilizes CAT tools to clean the data set noise and filter out repeated code-comment pairs.

[0012] Preferably, the screening in step 1 is performed using a large language model, and the hyperparameters of the large language model are adjusted.

[0013] Preferably, in step 2, the specific steps of generating the code summary are as follows:

[0014] Step 21, preprocessing the input code and parsing the input code into code fragments;

[0015] Step 22, extracting the code and its comments based on the context information and inputting them into the large language model;

[0016] Step 23, call the large language model to generate sentence-level code summaries and output them using the prompt strategy.

[0017] Preferably, the prompting strategies in step 2 include zero-sample prompting, few-sample prompting, thought chain prompting and reflective prompting.

[0018] The present invention provides a system for generating and automatically evaluating sentence-level code summaries based on a large language model, comprising:

[0019] Code summary dataset construction module: used to form a high-quality sentence-level code summary dataset, retaining code-comment pairs with high semantic relevance;

[0020] Code summary generation module: used to apply the prompt strategy to the code in the dataset to generate statement-level code summaries, and select the optimal code summary from the generated results;

[0021] Code summary evaluation module: used to evaluate the quality of code summaries;

[0022] The code summary evaluation module also includes a large language model evaluation, an indicator evaluation and a comprehensive evaluation module.

[0023] Preferably, the indicators of the indicator evaluation module include the BLEU indicator for measuring the accuracy of the generated results, the METEOR indicator for evaluating the semantic diversity and readability of the code summary, and the ROUGE indicator for reflecting the coverage of the code summary.

[0024] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. Multiple rounds of data cleaning are performed to reduce the noise data in the data set and improve the quality of the data set; 2. Through context enhancement technology and prompt strategies, the accuracy, context consistency and semantic integrity of code summary generation are improved; 3. The automated evaluation process greatly improves the evaluation efficiency and reduces the cost of manual evaluation; 4. By building an automated evaluation framework, the judgment results of the code summary are more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the main framework of the present invention;

[0026] Figure 2 This is a schematic diagram of LLM screening of the present invention;

[0027] Figure 3 This is a schematic diagram of a zero sample prompt of the present invention;

[0028] Figure 4 This is a schematic diagram of a small sample prompt of the present invention;

[0029] Figure 5 It is a schematic diagram of the reflective prompt of the present invention;

[0030] Figure 6 It is a schematic diagram of the thought chain prompt of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0032] (1) Constructing a code summary dataset

[0033] Through multimodal pre-training models and auxiliary tools, a high-quality statement-level code summary dataset is constructed, which specifically includes the following processes: extracting statement-level code and its corresponding comments from code snippets. For the comment part in the code, all multi-line comments are deleted, and then the code is processed line by line to identify single-line comments, ensuring that each comment can accurately correspond to the corresponding code statement; then use the improved CAT tool to clean the code and comments to remove invalid comments, duplicate code comments and inappropriate comments.

[0034] Finally, the large language model is used to analyze the semantic relevance between the remaining code and comments, and the code-comment pairs with high relevance are screened out through the binary classification task, and the low-relevance data is eliminated. In the example, the target statement code is ˋreturnresultˋ, the function is a function that calculates integer factorials, and the comment is ˋReturn the final computedfactorial valueˋ. The model evaluates the consistency and semantic matching between the comment and the target statement code by combining the function and context information of the statement, and outputs "low" or "high" according to the task requirements. In this example, the model judges that the comment accurately describes the function of the target statement, and finally outputs "high", indicating that the comment is highly relevant to the code. Through this method, the large language model can efficiently filter data, eliminate low-quality comments, and retain high-quality comments that are consistent with the code function, providing reliable support for building high-quality datasets.

[0035] (2) Generate code summary

[0036] First, the input code is preprocessed and the code file is parsed into statement-level code snippets. At the same time, the code context information is extracted, including other codes and comments in the function. At the same time, the indentation and format of the code are unified to ensure the standardization and consistency of the input data.

[0037] In terms of prompt strategy design, given the target code ˋreversed.append(input.charAt(i))ˋ, the model generates comments through four prompt strategies and uniformly evaluates and selects the best result. First, the zero-shot prompt is used to generate the comment `Appends a character to the reversed string.`; secondly, the few-shot prompt is used to provide a small number of high-quality examples, and the output is `Appends characters in reverse order to the reversedstring.`; then the thinking chain prompt is used to analyze the code logic step by step, and the function and statement analysis are combined to generate the comment `Appends each character from the input string in reverse order to the reversedobject.`; finally, the reflective prompt method is used to optimize the initial generation through reflection, and the comment `Addscharacters to the reversed object sequentially in reverse order.` is output.

[0038] By uniformly evaluating the results of the four hinting strategies, the generated comment `Appends characters in reverse order to construct the reversed string.` is selected as the best result.

[0039] (3) Evaluation code summary

[0040] After completing the code comment generation or modifying the existing comments, the quality of the comments needs to be evaluated. First, provide the complete function-level code as the context information of the statement-level code. For example, the function's input parameters, loop logic, or return value information provides the basis for the interpretation of the target statement. Provide the target statement-level code, which is the core code snippet for generating and evaluating comments, such as the code `reversed.append(input.charAt(i))` shown in the figure. Provide multiple candidate comments, which are generated through different prompting strategies and used for subsequent model and manual scoring evaluation tasks.

[0041] In the large language model evaluation stage, the large language model is called using predefined prompts to generate a semantic similarity score. The model outputs a higher similarity score, indicating that the comment has good semantic consistency with the code. In the indicator evaluation stage, the BLEU, METEOR and ROUGE indicator values ​​are calculated. BLEU measures the accuracy of the generated result by comparing the n-gram matching degree between the generated summary and the reference summary. METEOR combines stem matching and synonym matching to evaluate the semantic diversity and readability of the summary. ROUGE reflects the coverage of the summary by calculating the long common subsequence of the generated summary and the reference summary.

[0042] Among them, the BLEU indicator calculation formula is: BP stands for length penalty, which is used to prevent the generated text from being too short; exp stands for exponential function, which is used to smooth the summation result; N stands for the maximum order of n-gram; p n Indicates the accuracy of n-gram; w n Represents the weight of n-gram, usually divided equally:

[0043] The calculation formula of the METEOR indicator is: METEOR=F·(1-P), where F represents the weighted harmonic mean of precision and recall; P represents precision, which is the ratio of the number of matching words in the generated text to the total number of words in the generated text; R represents recall, which is the ratio of the number of matching words in the generated text to the total number of words in the reference text; (1-P) represents the penalty factor, which controls the impact of repeated words in the generated text.

[0044] The calculation formula of ROUGE indicator is: LCS stands for Longest Common Subsequence, which is the maximum continuous phrase with consistent order between the generated text and the reference text; Count(match) represents the number of matched words in LCS; Count(total) represents the total number of words in the reference text.

[0045] The indicator evaluation results are shown in the following table:

[0046] Model Tips Technology BLEU METEOR ROUGE-L Llama3.1 Zero sample prompt 10.2869 0.1575 0.1771 Few sample tips 9.4188 0.1466 0.1637 Reflective prompts 8.6626 0.1070 0.1351 Thought chain tips 8.8909 0.1508 0.1637 Gemma2 Zero sample prompt 8.9276 0.1689 0.1754 Few sample tips 9.2221 0.1614 0.1739 Reflective prompts 6.8018 0.1094 0.1218 Thought chain tips 8.1879 0.1812 0.1694 Gpt-4o-mini Zero sample prompt 9.3520 0.1649 0.1769 Few sample tips 9.4308 0.1717 0.1802 Reflective prompts 8.9755 0.1269 0.1474 Thought chain tips 8.3258 0.1279 0.1393 DeepSeek-Coder-V2 Zero sample prompt 6.6003 0.1349 0.1322 Few sample tips 8.8294 0.1465 0.1618 Reflective prompts 6.9696 0.1205 0.1260 Thought chain tips 5.2798 0.1333 0.1154

[0047] The large language model evaluation process receives function-level code, target statement code, and each candidate comment as input, analyzes the quality of each comment, and scores them from 1 to 5 according to the scoring criteria. The higher the score, the better the quality of the code summary.

[0048] The scoring criteria include conciseness: whether the comment is expressed in simple and clear language, avoiding the use of lengthy or complex sentences; accuracy: whether the comment accurately describes the core logic and function of the target statement, and whether there are semantic deviations or errors; abstraction level: whether the comment summarizes the code function at an appropriate level of abstraction, avoiding excessive description of specific implementation details, so as to facilitate developers to quickly understand the intent of the statement. The output format is such as `Comment1:5`, indicating that the first comment is scored 5 points.

[0049] Among them, CV represents the coefficient of dispersion, which is used to compare the degree of dispersion of different prompting technologies and reflect the stability of the results. A smaller coefficient of dispersion indicates that the results generated by the corresponding prompting technology are more stable; and a larger coefficient of dispersion indicates that the results generated by the corresponding prompting technology are more unstable.

[0050] The results of the large language model evaluation are shown in the following table:

[0051] Model Tips Technology Score CV(%) Llama3.1 Zero sample prompt 4.065 22.34 Few sample tips 4.135 21.09 Reflective prompts 3.665 25.70 Thought chain tips 3.935 23.22 Gemma2 Zero sample prompt 4.430 18.32 Few sample tips 4.335 17.50 Reflective prompts 4.070 21.40 Thought chain tips 4.160 21.07 Gpt-4o-mini Zero sample prompt 4.615 13.68 Few sample tips 4.645 13.63 Reflective prompts 4.150 25.17 Thought chain tips 3.730 21.97 DeepSeek-Coder-V2 Zero sample prompt 3.395 30.53 Few sample tips 3.860 23.52 Reflective prompts 3.755 24.97 Thought chain tips 3.695 31.31

[0052] During the manual scoring stage, in order to ensure the fairness and reliability of the scoring, experienced experts in the field are invited to manually score the annotations. The experts follow the same scoring criteria, conduct a comprehensive review from language expression, functional summary to practicality, and score each annotation in an independent manner. The manual scoring results serve as a high-credibility benchmark to provide a reference for subsequent consistency analysis.

[0053] The scoring criteria range from 1 to 5, and the higher the score, the better the quality of the code summary.

[0054] The results of manual evaluation are shown in the following table:

[0055] Model Tips Technology Score CV(%) Llama3.1 Zero sample prompt 4.050 21.69 Few sample tips 4.125 19.12 Reflective prompts 3.605 25.70 Thought chain tips 3.970 23.81 Gemma2 Zero sample prompt 4.635 14.21 Few sample tips 4.495 15.92 Reflective prompts 4.175 21.13 Thought chain tips 4.360 19.51 Gpt-4o-mini Zero sample prompt 4.720 12.41 Few sample tips 4.740 12.73 Reflective prompts 3.995 27.10 Thought chain tips 3.630 27.10 DeepSeek-Coder-V2 Zero sample prompt 3.610 25.11 Few sample tips 3.915 22.29 Reflective prompts 3.700 23.62 Thought chain tips 3.895 24.60

[0056] Then, through consistency analysis and verification, the weighted Kappa consistency analysis method is used to measure the matching degree between the large language model score and the manual score. The Kappa value ranges from 0 to 1. A Kappa value close to 1 indicates that the score is highly consistent and the large language model has good reliability. A Kappa value close to 0 indicates low consistency, suggesting that the model prompt or scoring strategy needs to be further optimized. The Kappa formula is as follows:

[0057]

[0058] In the formula, K represents the weighted Kappa value, which ranges from 0 to 1, with 1 representing perfect agreement, 0 representing agreement comparable to random agreement, and negative values ​​representing agreement below the random level; W in the formula ij It is a weight matrix used to measure the severity of the score difference, where i and j are the values ​​of the two scores, k is the total number of score levels, and when i=j, the weight is 0, indicating complete consistency. The greater the difference, the higher the weight. The numerator calculates the weighted total difference of the observed score distribution, and the denominator calculates the weighted total difference of the expected score distribution under random conditions. Weighted Kappa quantifies the score consistency by comparing the ratio of the numerator and denominator, taking into account the impact of score differences on consistency, making the evaluation more detailed and accurate.

[0059] This analysis process not only quantifies the performance of the model in complex annotation evaluation tasks, but also provides a scientific basis for the collaborative work between artificial intelligence and experts. The performance results of the consistency analysis are shown in the following table:

[0060] Model Tips Technology Weighted Kappa Score Llama3.1 Zero sample prompt 0.600 Few sample tips 0.542 Reflective prompts 0.523 Thought chain tips 0.543 Gemma2 Zero sample prompt 0.429 Few sample tips 0.494 Reflective prompts 0.562 Thought chain tips 0.488 Gpt-4o-mini Zero sample prompt 0.553 Few sample tips 0.487 Reflective prompts 0.610 Thought chain tips 0.484 DeepSeek-Coder-V2 Zero sample prompt 0.537 Few sample tips 0.535 Reflective prompts 0.453 Thought chain tips 0.405 average value 0.515

[0061] Finally, the comprehensive evaluation module combines the large language model score with the indicator evaluation results to generate a unified comprehensive evaluation score, and outputs a detailed evaluation report containing each score, the large language model score and the overall evaluation. If the evaluation results show that the annotation quality is high, the subsequent code integration and testing work can continue; if there are problems, the annotations are optimized and adjusted according to the prompts in the evaluation report until the annotation quality meets the requirements.

Claims

1. A method for generating and automatically evaluating statement-level code summaries based on a large language model, characterized in that: The following steps are involved: (1) Constructing a code summary dataset: extracting codes and their corresponding comments from code snippets, constructing code-comment pairs, and performing data cleaning and screening. All code-comment pairs constitute a code summary dataset; (2) Generate code summaries: Use a large language model, combined with prompt strategies and context enhancement techniques, to generate sentence-level code summaries from the codes in the code summary dataset; (3) Evaluate code summaries: Combine the large language model evaluation and indicator evaluation results to generate a comprehensive evaluation score for the code summary.

2. The method for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 1, characterized in that: The data cleaning described in step 1 includes removing all multi-line comments, splitting the code line by line and identifying single-line comments.

3. The method for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 1, characterized in that: In the data cleaning described in step 1, entries containing import or package keywords are removed from non-functional codes.

4. The method for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 1, characterized in that: The data cleaning described in step 1 includes limiting the comments to contain at least 3 words and only English characters or numbers, and filtering out comments with incomplete syntax.

5. The method for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 1, characterized in that: In step 1, data cleaning is performed using CAT tools to clean the data set noise and filter out duplicate code-comment pairs.

6. The method for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 1, characterized in that: The screening described in step 1 uses a large language model for screening, and adjusts the hyperparameters of the large language model.

7. The method for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 1, characterized in that: In step 2, a code summary is generated, and the specific steps are as follows: Step 21, preprocessing the input code and parsing the input code into code fragments; Step 22, extracting the code and its comments based on the context information and inputting them into the large language model; Step 23, call the large language model to generate sentence-level code summaries and output them using the prompt strategy.

8. The method for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 1, characterized in that: The prompting strategies described in step 2 include zero-sample prompting, few-sample prompting, thought chain prompting and reflective prompting.

9. A sentence-level code summary generation and automatic evaluation system based on a large language model, characterized in that: include: Code summary dataset construction module: used to form a high-quality sentence-level code summary dataset, retaining code-comment pairs with high semantic relevance; Code summary generation module: used to apply the prompt strategy to the code in the dataset to generate statement-level code summaries, and select the optimal code summary from the generated results; Code summary evaluation module: used to evaluate the quality of code summaries; The code summary evaluation module also includes a large language model evaluation, an indicator evaluation and a comprehensive evaluation module.

10. The system for generating and automatically evaluating statement-level code summaries based on a large language model according to claim 9, characterized in that: The indicators of the indicator evaluation module include the BLEU indicator for measuring the accuracy of the generated results, the METEOR indicator for evaluating the semantic diversity and readability of the code summary, and the ROUGE indicator for reflecting the coverage of the code summary.

Images