README document automatic generation method for code warehouse

Abstract

The invention discloses an automatic README document generation method for a code repository, which comprises the following steps of: firstly, carrying out importance sorting on code files in the code repository by adopting a pre-trained machine learning model, and automatically screening out the most critical code files, so that the problems of inaccuracy and low efficiency caused by manual screening are avoided; therefore, pertinence and accuracy of document generation are improved. Secondly, the key code file is subjected to function-level splitting, and the function abstract is generated based on the control flow and the data flow, so that the document content not only covers the function of the code, but also embodies the complexity of the code logic and structure, and the document is ensured to comprehensively and accurately describe the core function of the code. And finally, the README document is generated by utilizing the large language model, so that the automatic writing of the document is further realized, the workload of manual intervention is reduced, and meanwhile, the generated document is ensured to have natural and smooth language expression and consistency.

Description

Technical Field

[0001] The present invention belongs to the cross-technical field of artificial intelligence and software engineering, and specifically relates to a method for automatically generating a README document for a code repository. Background Art

[0002] In modern software development, README documents are an important part of the code repository. They provide a project introduction, main function descriptions, installation and usage guides, code structure overviews, and explanations of key codes. For developers, README documents can help them quickly understand and get started with the project. However, manually writing and maintaining README documents is a time-consuming and laborious task, especially for large and constantly changing code bases, which leads to the need for automatically generating README documents.

[0003] As the code repository grows in size and complexity, manually writing README documents is not only inefficient, but also prone to omissions or inaccurate information. Therefore, it is particularly important to use advanced technical means to automatically generate README documents; through automation, not only can the content of the README document be ensured to always be consistent with the code repository, but it can also reduce human errors, improve document quality and development efficiency.

[0004] Existing methods for assisting developers in writing README documents mainly rely on providing static templates or filling in templates based on user input. Such tools include GitHub's default README generator and other static document generation tools. These tools usually use predefined templates to let developers manually fill in content according to project requirements, such as project name, instructions, installation steps, etc. However, these solutions cannot actually help developers write specific document content. Due to the limitations of template generation, developers still need to spend a lot of time and energy to write detailed project introductions, function descriptions, instructions, etc. Especially for complex projects, this method is not only inefficient, but also easily leads to uneven document quality.

[0005] In addition, existing studies have shown that large language models have the ability to generate class / function-level code comments. For example, the literature [Chen, Mark, Jerry Tworek, etc. "Evaluating Large Language Models Trained on Code." ArXiv abs / 2107.03374 (2021): n.pag] uses language models to generate function comments, and generates corresponding comment text by parsing and summarizing the input parameters, output results, and internal logic of the function; this method can provide more accurate comment assistance for a small range of code, thereby improving the readability and maintainability of the code. However, the application scope of such methods is limited to the generation of comments at the function or class level, which usually only involves the parsing and generation of dozens of lines of code.

[0006] The challenge of generating README documents is that they require a high degree of abstraction and generalization of the entire project. README documents usually need to cover the overall information of the project, including project goals, main functional modules, installation steps, usage methods, project structure, etc., involving thousands or even millions of lines of code. Therefore, the method of generating comments based solely on existing large language models is not enough to cope with the complexity of README document generation; existing technologies cannot perform overall abstraction from the project level, and the generated content often lacks systematicity and globality.

[0007] Existing methods such as code documentation generation tools (such as Sphinx and Doxygen) can automatically generate API documentation based on code comments, but their role is limited to extracting documents from the code itself, and they cannot automatically generate README documents that summarize the entire project; these tools usually rely on comments written manually by developers and cannot automatically generate project-level documents. Due to the lack of in-depth understanding of the global structure and logic of the code, the documents generated by such methods are also difficult to meet the developers' needs for high-level documents. The current optimal method LARCH [Koreeda, Yuta & Morishita, Terufumi & Imaichi, Osamu & Sogawa, Yasuhiro. (2023). LARCH: Large Language Model-based Automatic Readme Creation with Heuristics. 10.48550 / arXiv.2308.03099] extracts the entry file of the project through a heuristic method and then generates a README document, but this method also has very obvious limitations, because the entry of the project cannot represent the meaning of the entire project.

[0008] It can be seen that the existing methods have significant limitations: on the one hand, the static template filling method lacks automation capabilities and cannot help developers generate specific content; on the other hand, the technology of generating comments based on language models can only handle a small range of code snippets and cannot effectively meet the document generation needs of large-scale code bases. Therefore, how to generate a complete, accurate and abstract README document at the project level is still a technical problem that needs to be solved urgently. Summary of the invention

[0009] In order to meet the above challenges, the present invention provides a method for automatically generating a README document for a code repository, which can provide developers with a powerful tool for document writing for complex code repositories.

[0010] A method for automatically generating a README document for a code repository includes the following steps:

[0011] (1) Use pre-trained machine learning models to extract key code files from the code repository;

[0012] (2) Split key code files at the function level to generate function summaries;

[0013] (3) The metadata of the code repository and the function summary of the key code files are used as prompt words and input into the large language model to automatically generate the README document.

[0014] Furthermore, the specific implementation of step (1) is as follows:

[0015] 1.1 Obtain the dataset of the code repository;

[0016] 1.2 Generate label data for each code file in the dataset and determine the importance of the code file based on the label data;

[0017] 1.3 Extract features from the code files and use them as input to the machine learning model, and use the importance as the output of the model to train the model;

[0018] 1.4 Use the trained model to sort the importance of the code files in the code repository, and select several code files with high rankings as the key code files of the entire code repository.

[0019] Furthermore, in step 1.1, a code repository dataset is obtained, that is, an API request is constructed according to the programming language, number of stars, and size limit of the specified repository, and a paging query is used to bypass the access restriction of the API request to obtain a large number of code repositories and their metadata as a dataset.

[0020] Furthermore, in step 1.2, label data is generated for each code file in the data set, that is, a set of heuristic label functions are artificially created to create noisy label data for each code file using the paradigm of data programming, and then the statistical model of the snorkel library is used to handle label conflicts, and the accuracy of the label function is restored by solving the matrix completion problem, and then the label data output from different label functions are integrated to obtain the importance of the code file.

[0021] Furthermore, the label function includes the following four types of information judgment:

[0022] File information, including whether the file length is too short (less than 200 characters), whether the function in the file contains the "main" string, whether the file contains a parameter parser, etc.

[0023] Folder information, including whether the file name contains the string "main", whether it contains the name of the program entry class (such as cli.py), etc.

[0024] Static code analysis information, including whether it is at the top level of the import dependency tree, whether it contains classes that are inherited multiple times, etc.

[0025] Oracle information, including whether it is referenced in the original README document, whether it is listed in setup.py, etc.

[0026] Furthermore, the specific implementation of step (2) is as follows:

[0027] 2.1 Split the key code files at the function level to obtain the control flow and data flow of the function;

[0028] 2.2 Extract data flow information and control flow information in each function based on the abstract syntax tree;

[0029] 2.3 Generate a summary for each function based on the data flow information and control flow information.

[0030] Furthermore, in step 2.1, the key code file is split at function level, that is, the string of the key code file is parsed into an abstract syntax tree, the abstract syntax tree is traversed to obtain the definition and code of each function, and then the data flow and control flow of each function are obtained through the attributes of the node.

[0031] Furthermore, in step 2.3, the GPT-4 API is called based on the data flow information and control flow information to generate a summary for each function. Specifically: first, the function's data flow information and control flow information (including input parameters, return values, key operations and conditional branches in the function body, etc.) are used as input data of the GPT-4 API; then these input data are constructed into a formatted prompt word and passed to the GPT-4 API, requesting the generation of a detailed summary. The GPT-4 API will generate a summary covering all aspects of the function based on its pre-trained language model and the provided context information, including the function's input parameters, output results, key logic, and potential boundary conditions.

[0032] Furthermore, the specific implementation of step (3) is as follows:

[0033] 3.1 Use the metadata of the code repository and the function summary of key code files as prompts and format them;

[0034] 3.2 Build an API request and send it to the server where the large language model is located. The API request includes authentication information (such as API key), formatted prompt words, and other parameter settings that may affect the generation results (such as the length of the generated text, temperature parameters, etc.);

[0035] 3.3 The server parses the received API request, inputs the formatted prompt words into the large language model, and uses the function summary of the key code file as the context of the model input to generate the README document.

[0036] The present invention significantly improves the generation efficiency and quality of README documents in the code warehouse by combining a pre-trained machine learning model and a large language model, and has important technical effects. First, the present invention uses a pre-trained machine learning model to sort the importance of code files in the code warehouse, and can automatically screen out the most critical code files, avoiding the inaccuracy and inefficiency caused by manual screening, thereby improving the pertinence and accuracy of document generation. Secondly, the present invention performs function-level splitting on key code files and generates function summaries based on control flow and data flow, so that the document content not only covers the functions of the code, but also reflects the complexity of the code logic and structure, ensuring that the document can fully and accurately describe the core functions of the code. This deep code analysis greatly improves the accuracy of the document content. Finally, the present invention uses a large language model to generate a README document, further realizing the automatic writing of the document, reducing the workload of manual intervention, and ensuring that the generated document has natural and fluent language expression and consistency; this technical effect enables developers to quickly generate high-quality documents, significantly improving the efficiency of software development, especially in the maintenance and collaboration of large code warehouses, and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The figure is a flow chart of the method for automatically generating a README document according to the present invention.

[0038] Figure 2 Schematic diagram of prompt words in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0040] like Figure 1 As shown, the method for automatically generating a README document for a code repository of the present invention comprises the following steps:

[0041] (1) Key code extraction.

[0042] S11. Obtaining the code repository dataset: Build an API request and filter the code repositories according to the following conditions: ① The number of stars exceeds 100; ② Use Python as the programming language; ③ Less than 100MB; ④ Created after July 1, 2022; Use paginated queries to bypass API access restrictions, select the 1,000 repositories with the highest number of stars every six months, obtain their metadata and clone these repositories locally.

[0043] S12. Label function and feature design: We use the data programming paradigm and weak supervision methods to manually create a set of heuristic label functions to create pseudo-label data. The specific label functions include four categories:

[0044] File information: whether the file length is too short (less than 200 characters), whether the function in the file contains the "main" string, whether the file contains a parameter parser, etc.

[0045] Folder information: whether the file name contains the "main" string, whether it contains the name of the program entry class (for example, cli.py), etc.

[0046] Static code analysis information: whether it is at the top level of the import dependency tree, whether it contains classes that are inherited multiple times, etc.

[0047] Oracle information: whether it is referenced in the original README document, whether it is listed in setup.py, etc.

[0048] After obtaining noisy labels through this series of labeling functions, the labeling model of the snorkel library is used to handle label conflicts, restore the accuracy of these labeling functions by solving the matrix completion problem, and integrate the outputs from different labeling sources.

[0049] S13. Importance sorting: The importance problem is defined as a ranking learning problem for files within each code base. A set of features similar to the label function is designed, but unlike the label function, the feature values ​​are not completely Boolean values, which allows for greater numerical flexibility. After obtaining the features, these features are used to train the gradient boosting tree model, and the trained model is used to sort the files in the code repository. Finally, the top k ranked files are selected as the key code part of the entire repository.

[0050] (2) Static code analysis and summary.

[0051] S21. Function splitting: In order to understand the intent and logic of the code in a more fine-grained manner, the file code is split at the function level, that is, the source key code file string is parsed into an abstract syntax tree, the abstract syntax tree is traversed to obtain the definition and code of each function, and the corresponding data flow and control flow are obtained through the attributes of the node.

[0052] S22. Data flow and control flow extraction: Based on the parsed abstract syntax tree, extract the data flow and control flow information in each function to analyze the logic and behavior of the code in a more fine-grained manner.

[0053] S23. Summary generation: Based on the extracted data flow and control flow information, a summary of each function is generated by calling the GPT-4 API. These summaries will cover the function's input parameters, output results, key logic, and potential boundary conditions. They will serve as one of the inputs for automatic generation of the README document, providing a detailed explanation of the key code sections.

[0054] Specifically, first prepare the detailed information of the function, including input parameters, return values, key operations in the function body, and conditional branches, as input data; then, build this data into a formatted prompt word and pass it to the GPT-4 API to request the generation of a detailed summary. GPT-4 will generate a summary covering all aspects of the function based on its pre-trained language model and the provided contextual information, ensuring a comprehensive understanding and description of the code.

[0055] (3) README document is automatically generated.

[0056] S31. Prompt word design: The metadata of the code repository and the summary generated by the key code are used as prompt words. The metadata includes information such as the repository name and author. The key code summary includes the main functions and uses of each module or function. This information will be combined into a detailed prompt word such as Figure 2 As shown, a large language model is provided to ensure that the generated README document covers all important aspects.

[0057] S32. Contextual Learning: In order to generate a coherent and factually correct README document, it is necessary to generate the summary R of the key code as the context of the input of the large language model. j} j≤i In the context of , the large language model can generate a symbol sequence by calculating the probability of the i+1th symbol; taking the summary R as the symbol sequence {r i}, generating a symbol sequence {x0,…,x n The probability of} is as follows:

[0058]

[0059] S33. GPT-4 API call: In this implementation, the large language model uses the GPT-4 API, so the designed prompt words are converted into the format required for the API request, the request data is sent to the GPT-4 API, and the final generated README document is obtained. The document includes a project introduction, main function description, installation and usage guide, code structure overview, and explanation of key codes, which can help new developers quickly get started and understand the project. The specific calling process of the GPT-4 API in this implementation is as follows:

[0060] ① Data formatting: Format the designed prompt words, including the metadata of the code repository and the summary generated by the key code, into an appropriate input form according to the requirements of the GPT-4 API. The formatted content needs to ensure a clear structure and complete information so that GPT-4 can accurately understand and process it.

[0061] ②Build API request: Use an appropriate programming language (such as Python) to build an API request. The request needs to include necessary authentication information (such as API key), formatted prompt words, and other parameter settings that may affect the generation results (such as the length of the generated text, temperature parameters, etc.).

[0062] ③ Send request: Send the constructed API request to the GPT-4 server, and use the HTTP request library (such as requests) to implement this process; when sending the request, it is necessary to handle possible network errors or timeouts to ensure that the request can be completed smoothly.

[0063] ④ Receive and process responses: After receiving the response data returned by GPT-4, parse the response content, extract the text in the generated README document, and perform necessary post-processing on the generated text, such as format adjustment, spelling check, etc., to ensure that the final README document is formatted correctly and the content is complete.

[0064] ⑤Store and use the generated README document: Save the processed README document to a suitable location, such as the root directory of the code repository; as needed, you can further push the README document to a remote repository or integrate it with other automated document generation tools.

[0065] In traditional text generation tasks, word overlap-based metrics are very commonly used, but word overlap-based metrics such as BLEU and Rouge perform poorly in evaluating the quality of README document generation, because the task of generating README documents may have multiple reasonable reference answers. Word overlap metrics usually rely on a single reference text and cannot handle the diversity of reference answers, which means that even if the generated README document is reasonable in content, these metrics will give a low score if it does not match a single reference text.

[0066] In order to test the effect of automatic generation of README documents of the present invention, we use a large model to evaluate the quality of the generated README documents. Specifically, this embodiment uses GPT-4, an advanced language model, as a discriminator to compare the effects of different methods. In order to avoid too many requests for GPT-4 API, we randomly selected 50 pairs from the data set as approximations. We not only compared the BLEU and Rouge scores obtained by comparing the current optimal README generation method LARCH and the method of the present invention with the original README document in terms of generation quality, but also used the WinRate obtained by pairwise comparison as a judgment indicator. The final comparison results are shown in Table 1:

[0067] Table 1

[0068]

[0069] As can be seen from the table, the BLEU and Rouge scores of the present invention are slightly lower than those of the LARCH method (LARCH's BLEU is 0.0183 and Rouge is 0.162, while the BLEU of the present invention is 0.0169 and Rouge is 0.143), but in the Win-Rate indicator, the score of the present invention is 0.68, which is significantly higher than LARCH's 0.64. This shows that although the score of the present invention is slightly lower in the traditional text generation quality evaluation indicators such as BLEU and Rouge, in the actual user experience (measured by Win-Rate), the README document generated by the present invention is more popular with users and shows better actual effects; this may be because the method of the present invention has better performance in the overall structure, content relevance or user-friendliness of the document.

[0070] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A method for automatically generating a README document for a code repository, comprising the following steps: (1) Use pre-trained machine learning models to extract key code files from the code repository; (2) Split key code files at the function level to generate function summaries; (3) The metadata of the code repository and the function summary of the key code files are used as prompt words and input into the large language model to automatically generate the README document.

2. The method for automatically generating a README document according to claim 1, wherein: The specific implementation of step (1) is as follows: 1.1 Obtain the dataset of the code repository; 1.2 Generate label data for each code file in the dataset and determine the importance of the code file based on the label data; 1.3 Extract features from the code files and use them as input to the machine learning model, and use the importance as the output of the model to train the model; 1.4 Use the trained model to sort the importance of the code files in the code repository, and select several code files with high rankings as the key code files of the entire code repository.

3. The method for automatically generating a README document according to claim 2, characterized in that: In the step 1.1, the code repository dataset is obtained, that is, an API request is constructed according to the programming language, number of stars, and size limit of the specified repository, and a paging query is used to bypass the access restrictions of the API request to obtain a large number of code repositories and their metadata as a dataset.

4. The method for automatically generating a README document according to claim 2, wherein: In step 1.2, label data is generated for each code file in the data set, that is, a set of heuristic label functions are artificially created to create noisy label data for each code file using the paradigm of data programming, and then the statistical model of the snorkel library is used to handle label conflicts, and the accuracy of the label function is restored by solving the matrix completion problem, and then the label data output from different label functions are integrated to obtain the importance of the code file.

5. The method for automatically generating a README document according to claim 4, characterized in that: The label function includes the following four types of information judgment: File information, including whether the file length is too short, whether the function in the file contains the "main" string, and whether the file contains a parameter parser; Folder information, including whether the file name contains the "main" string and whether it contains the name of the program entry class; Static code analysis information, including whether it is at the top level of the import dependency tree and whether it contains classes that are inherited multiple times; Oracle information, including whether it is referenced in the original README document and whether it is listed in setup.py.

6. The method for automatically generating a README document according to claim 1, wherein: The specific implementation of step (2) is as follows: 2.1 Split the key code files at the function level to obtain the control flow and data flow of the function; 2.2 Extract data flow information and control flow information in each function based on the abstract syntax tree; 2.3 Generate a summary for each function based on the data flow information and control flow information.

7. The method for automatically generating a README document according to claim 6, characterized in that: In step 2.1, the key code file is split at function level, that is, the string of the key code file is parsed into an abstract syntax tree, the abstract syntax tree is traversed to obtain the definition and code of each function, and then the data flow and control flow of each function are obtained through the attributes of the node.

8. The method for automatically generating a README document according to claim 6, characterized in that: In step 2.3, the GPT-4 API is called based on the data flow information and control flow information to generate a summary for each function. Specifically: first, the function's data flow information and control flow information are used as input data of the GPT-4 API; then these input data are constructed into a formatted prompt word and passed to the GPT-4 API, requesting the generation of a detailed summary. The GPT-4 API will generate a summary covering all aspects of the function based on its pre-trained language model and the provided context information, including the function's input parameters, output results, key logic, and potential boundary conditions.

9. The method for automatically generating a README document according to claim 1, wherein: The specific implementation of step (3) is as follows: 3.1 Use the metadata of the code repository and the function summary of key code files as prompts and format them; 3.2 Build an API request and send it to the server where the large language model is located. The API request includes authentication information, formatted prompt words, and other parameter settings that may affect the generation results; 3.3 The server parses the received API request, inputs the formatted prompt words into the large language model, and uses the function summary of the key code file as the context of the model input to generate the README document.

Images