A version-controllable dataset automatic evolution method based on a large language model

Abstract

The application discloses a version-controllable dataset automatic evolution method based on a large language model, mainly including a data crawling module, a data analysis module, a data labeling module and a task form processing module. The application reduces the need for manual intervention by introducing an automatic updating mechanism. Specifically, the multi-agent system can monitor the version update of third-party libraries in the Python ecosystem and update the dataset according to the latest library version. This automatic updating mechanism not only improves the efficiency of dataset maintenance, but also ensures the timeliness of the dataset, providing reliable technical support for the adaptability evaluation of LLMs in dynamic development environments.

Description

Technical Field

[0001] This invention relates to the field of code generation, and more particularly to an automatic evolution method for version-controlled datasets based on large language models. Background Technology

[0002] In recent years, large language models, such as ChatGPT and CodeLLaMA, have made significant breakthroughs in code generation, with their performance evaluation primarily relying on static benchmark datasets like HumanEval and MBPP. However, software development is inherently dynamic and evolving, manifested in frequent version updates of software libraries and continuous iterations of API interfaces.

[0003] Traditional evaluation methods have significant limitations: existing datasets fail to accurately simulate the dynamic changes in library version dependencies in real-world development scenarios, making it impossible to effectively test the adaptability of LLMs to specific library versions. VersiCode, a benchmark dataset specifically designed to evaluate the code generation capabilities of LLMs under version control, covers 300 Python libraries and over 2000 versions, providing a comprehensive and realistic testing platform for version-aware code generation by LLMs.

[0004] Furthermore, LLMs exhibit reasoning and planning capabilities comparable to humans, leading to the rapid development of LLM-based agents. These agents can understand and generate human-like instructions, supporting complex interactions and decision-making. To further enhance the capabilities of these agents, researchers have proposed LLMs-based multi-agent systems, which use multiple specialized agents to collaborate in solving complex problems. Compared to single-agent systems, multi-agent systems can simulate cooperation and competition in complex environments, demonstrating stronger task-solving abilities and finding wide application in software development, robotics systems, social simulation, policy making, and games.

[0005] As Python software libraries and APIs continue to evolve, ensuring that version-controlled datasets like VersiCode reflect the latest library updates and API changes is crucial for accurately evaluating the performance of LLMs in dynamic development environments. Traditional dataset maintenance often relies on manual updates, which is not only time-consuming and labor-intensive but also prone to missing important updates, affecting the accuracy of model performance evaluation. To address this issue, automated evolutionary methods based on large language models and multi-agent systems have emerged as a cutting-edge solution.

[0006] Traditional code generation datasets rely on manual maintenance, which is inefficient and difficult to control in terms of quality, affecting the accuracy and timeliness of evaluations. Furthermore, VersiCode, as the first benchmark dataset specifically designed for dynamic changes in Python library versions, faces challenges in maintaining its functionality due to the inefficiency of manual updates and the inability to reflect the latest library version changes in real time. Therefore, there is an urgent need to extend and enrich VersiCode through automation to improve its dynamic update capabilities.

[0007] Existing evaluation benchmarks have certain limitations: (1) Dataset construction is highly dependent on manual work, requiring experts to write tasks, test cases, and standard answers, which is time-consuming, labor-intensive, and has poor scalability. The quality of the dataset is easily inconsistent due to the subjective preferences of the designers; (2) Version information of the code is ignored, and the version differences of Python third-party libraries (such as API changes of pandas or numpy) are not considered. It is difficult to evaluate the performance of the model in generating code under a specific library version environment, and it is difficult to reflect version control issues in actual engineering scenarios. These problems cause the evaluation results to deviate from the actual application effect, and may overestimate the practicality of the model in the real environment.

[0008] This invention aims to solve the following key problems: (1) How to achieve automated maintenance of the VersiCode version controllable dataset, reduce manual intervention, and ensure that the dataset keeps pace with the latest and more popular library versions and API changes; (2) How to further expand and enrich the API evolution types of its Python library on the basis of the existing VersiCode benchmark dataset, so that it can more comprehensively reflect the changes in library version APIs, so as to accurately evaluate the code generation capabilities of LLMs in dynamic development environments; (3) How to improve the efficiency and flexibility of VersiCode dataset maintenance through multi-agent collaboration, and support the adaptive evaluation of LLMs in rapidly changing software development environments. Summary of the Invention

[0009] To address the aforementioned issues, this invention discloses a method for the automatic evolution of version-controlled datasets based on large language models. Through the collaborative work of multiple agents, it achieves automated monitoring of third-party library version updates and dataset updates in the Python ecosystem, effectively improving the maintenance efficiency and accuracy of datasets and providing reliable technical support for evaluating the dynamic environmental adaptability of LLMs.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] An automatic evolution method for version-controlled datasets based on large language models is proposed, which mainly includes a data crawling module, a data parsing module, a data annotation module, and a task format processing module. The specific contents of each module are as follows:

[0012] (1) Data crawling module

[0013] This module automates the collection of Python library version metadata from multiple data sources (such as GitHub and PyPI), ensuring the completeness and timeliness of the collected data, and performs initial data filtering. The process steps of this module are denoted as S1, and their specific steps are as follows:

[0014] In step S101, a web crawler script is used to search for legitimate Python libraries published on the PyPI website and obtain a list of library names.

[0015] In step S102, based on the list of library names obtained in step S101, the project of the Python library in the list is retrieved on the Github website to obtain meta information, including the number of stars of the project, release time, etc.

[0016] In step S103, based on the metadata obtained in step S102, Python libraries whose star count meets a threshold condition are filtered out and used as the objects to be crawled to construct the dataset. The threshold is manually set, such as a project with more than 10,000 stars or a star count increase of more than 1,000 in the past month. After filtering, a list of Python libraries to be crawled is obtained;

[0017] In step S104, based on the list of libraries to be crawled obtained in step S103, the libraries to be crawled are classified according to the Python library versions contained in the local VersiCode dataset: if the local library already exists in the list of libraries to be crawled, then the new version source code package of the library other than the local version is crawled; if it does not exist, then all historical version source code packages of the library within the specified time range are crawled.

[0018] In step S105, the metadata of the two types of Python libraries obtained in step S104, including library name, version number, release time, and the requires_python field (the Python version environment required to install the library version), is organized and handed over to the data parsing module for subsequent data processing.

[0019] (2) Data parsing module

[0020] This module automates the structuring and initial cleaning of crawled data, extracts key fields, and standardizes data formats to ensure data consistency and accuracy, providing reliable input for subsequent annotation and processing. The workflow of this module is denoted as S2, and its specific steps are as follows:

[0021] In step S201, the AexPy tool proposed in the literature "AexPy: Detecting API Breaking Changes in PythonPackages" is optimized. AexPy is a tool for detecting destructive changes to Python third-party library APIs. Its core functions involve (1) crawling Python library metadata, (2) extracting APIs within the library, (3) detecting API changes between library versions, and (4) generating API change detection reports. The four modules corresponding to these functions are "preprocess", "extract", "diff", and "report". Among them, the "preprocess" module is used to obtain Python library metadata, etc., and the "extract" module mainly uses the three built-in Python libraries importlib, pkgutil, and inspect to dynamically extract API information. The modifications to AexPy are as follows:

[0022] (1) The original AexPy can only run in the Python 3.12 environment. Modify the internal code of AexPy to make it support multiple versions of Python environment (such as Python 3.6 to 3.12);

[0023] (2) When AexPy extracts library version APIs, pip defaults to installing .whl files compatible with the latest Python version (e.g., 3.12) supported by the library. However, this can lead to incompatibility between dependency versions. For example, after installing a library, pip might select the latest version of the dependency library compatible with Python 3.12, resulting in version conflicts between the installed library and its dependencies. By parsing the `requires_python` field of the library version metadata to obtain the specified Python version and passing it to the AexPy tool, the problem of incompatibility between library and dependency versions can be effectively alleviated.

[0024] (3) When piping some Python libraries, the requirements.txt file may not contain all the required dependencies, causing the Extract module to fail to extract the API due to missing dependencies. By modifying the internal code of AexPy, the missing dependencies can be automatically identified and installed using a predefined library-dependency mapping relationship, thereby effectively mitigating the extraction failure problem caused by missing dependencies.

[0025] In step S202, based on the Python library metadata obtained in step S1, the `requires_python` field corresponding to each library version is parsed to obtain the Python version environment required to install that library version. The parsing method is based on regular expressions. For example, the value of `requires_python` is ">=3.6.1" or ">3.7", and the version number pattern is denoted as `major.minor.patch`. Regular expressions are written to match this type of pattern and capture the version number in the matched items. If it is ">=", the parsed Python version number is itself; if it is ">", if it contains the `patch` part, it returns `major.minor.(patch+1)`; if it does not contain `patch`, it returns `major.(minor+1)`.

[0026] In step S203, based on the Python version to be installed obtained in step S202, the preprocess function module of the AexPy tool is used to crawl the source code package of the corresponding Python version library for subsequent extraction of APIs from the library.

[0027] In step S204, based on the library version source code package crawled in step S203, the extract function module of the AexPy tool is used to extract APIs. The workflow of the AexPy tool is as follows: First, a virtual environment is created based on the executable Python version of the current library version, and the previously crawled source code package is installed into this virtual environment. Then, APIs from third-party Python libraries are extracted based on three built-in Python libraries: importlib, pkgutil, and inspect. importlib is used for dynamically loading modules, supporting the import of target libraries and their submodules on demand. pkgutil is responsible for traversing the package structure, recursively obtaining all modules and submodules in the library. inspect is used to extract information such as functions and classes from modules and obtain their metadata (such as parameter signatures and docstrings). At this point, the APIs and related information of the current library version can be obtained, recorded as an internal knowledge base of the library version, and stored in JSON format.

[0028] In step S205, based on the library version's internal knowledge base obtained in step S204, and following the Python API evolution pattern proposed in the literature "How Do Python Framework APIs Evolve? An Exploratory Study," a written script tool is used to find APIs that have evolved between two consecutive library versions (i.e., the internal knowledge bases of the two library versions). Each or several functions in the script correspond to an API evolution rule mentioned in the aforementioned literature. Thus, the API evolution relationships between library versions can be obtained. Each pair of evolutionary relationships consists of the APIs corresponding to the old and new versions and their related information. These evolutionary relationships can be recorded as an API evolution knowledge graph between library versions and stored in JSON format.

[0029] (3) Data labeling module

[0030] This module plans to automatically label the parsed data using an LLM-based agent and preset rules. The labeling strategy is adjusted according to different task scenarios, ensuring the accuracy of data labeling and laying the foundation for subsequent task processing and evaluation. The process steps of this module are denoted as S3, and their specific steps are as follows:

[0031] In step S301, based on the evolved APIs and related information obtained in step S20, code usage examples are constructed for these APIs as ground truth in the dataset to measure the correctness of the answers generated by the large model. An LLM-based agent is used to extract code usage examples from the API metadata docstring. The extraction method involves writing possible code example patterns from the docstring into prompts to inform the LLM agent of the context, thereby guiding it to identify and extract valid code snippets.

[0032] In step S302, based on the code snippets successfully extracted in step S301, a functional description is generated for them using an LLM-based agent. The role of the functional description is to be part of the dataset, along with relevant information such as the library version, and written into the prompt to inform the LLM agent as context. It can also be combined with methods such as few-shot learning to guide the generation of code for the corresponding version and function. In other words, this is a version-controlled code generation task.

[0033] In step S303, based on the data obtained in step S302, an automated script is used to perform data cleaning according to heuristic rules, and the successfully processed data is recorded as metadata. The metadata is in the form of a tuple, which contains at least three core elements: library version, function description, and code snippet. It may also contain other relevant information, such as API name.

[0034] (4) Task Format Processing Module

[0035] This module plans to transform labeled data into a form required for specific tasks through agents and automated scripts based on large language models, making it more suitable for the evaluation process, ensuring that the data meets the requirements of LLM performance evaluation scenarios, and improving the quality of the final dataset. The process steps of this module are denoted as S4, and their specific steps are as follows:

[0036] In step S401, based on the metadata obtained in step S3, the data is further processed into two task types: code completion and code migration, both contained in the VersiCode dataset. For the code completion task, targeting evolution types involving only a single library version, such as API additions and deletions, a script is used to locate the version-sensitive parts of each code snippet (i.e., the API names marked in the metadata) using an AST (Abstract Syntax Tree) package, and replace them with... <mask>The tags are used as the content to be inferred, allowing the LLM to predict the masked parts based on the context. For code migration tasks, specifically those involving cross-version evolution types, such as API renaming and relocation within Python libraries, a knowledge graph of API evolution between library versions is used. Each set of data involving two API versions serves as a test instance. The Python library name, source version, source version API name, and example code snippet using the source version are provided to the LLM, allowing it to infer that the target version contains the evolved API, thus achieving code migration from one version to another. The version before migration is called the source version, and the version after migration is called the target version.

[0037] The beneficial effects of this invention are:

[0038] (1) Automatically update the dataset, reduce manpower, and dynamically respond to library evolution;

[0039] This invention reduces the need for manual intervention by introducing an automated update mechanism. Specifically, the multi-agent system can monitor version updates of third-party libraries in the Python ecosystem and update the dataset in real time according to the latest library versions. This automated update mechanism not only improves the efficiency of dataset maintenance but also ensures the timeliness of the dataset, providing reliable technical support for the adaptive evaluation of LLMs in dynamic development environments.

[0040] (2) The API evolution pattern of the VersiCode dataset has been expanded;

[0041] While VersiCode is the first benchmark dataset specifically designed for dynamic changes in library versions, its coverage of Python API evolution patterns has limitations. This invention expands the scope of VersiCode's evolution patterns by introducing the AexPy tool and referencing richer API evolution patterns summarized in previous research, enabling it to more comprehensively reflect the actual situation of Python library version evolution. This invention automates the updating of the VersiCode dataset through multi-agent system collaboration, monitoring version updates of third-party libraries in the Python ecosystem in real time to ensure the dataset remains consistent with the latest library versions and API changes.

[0042] (3) Multi-agent dimension update of dataset improves efficiency and accuracy;

[0043] This invention achieves efficient dataset maintenance in a pipelined manner through the collaboration of a multi-agent system. Each agent is specifically designed for a particular task and executes it step by step in a predetermined sequence. Multiple agents work collaboratively in turn to complete the construction and updating of the dataset. For example, one agent builds code usage examples for a library version API obtained through automated script processing; then another agent generates corresponding functional descriptions for this code snippet to enrich the contextual information in the dataset that informs LLMs; and finally, a third agent generates test cases for each data instance. This multi-agent pipeline collaboration mechanism not only improves the efficiency of dataset maintenance but also enhances the accuracy and completeness of the dataset. Furthermore, the multi-agent system exhibits high flexibility and scalability, dynamically adjusting pipeline strategies according to the characteristics of different libraries and their APIs to ensure that the dataset remains consistent with the latest technological developments.

[0044] In summary, this invention significantly improves the usability, timeliness, and accuracy of the VersiCode dataset by extending its API evolution pattern, introducing automation mechanisms, and implementing multi-agent pipeline collaboration. This provides comprehensive and reliable technical support for evaluating the code generation capabilities of LLMs in dynamic development environments. Furthermore, this invention greatly reduces the need for manual intervention, providing technical assurance for efficient and long-term maintenance. Attached Figure Description

[0045] Figure 1 Flowchart of the data crawling module of this invention;

[0046] Figure 2 Flowchart of the data parsing module of this invention;

[0047] Figure 3 Flowchart of the data annotation module of this invention;

[0048] Figure 4 Flowchart of the task format processing module of this invention. Detailed Implementation

[0049] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to the directions in the accompanying drawings, and the terms "inner" and "outer" refer to the directions toward or away from the geometric center of a specific component, respectively.

[0050] An automatic evolution method for version-controlled datasets based on large language models is presented in the following implementation example:

[0051] Assuming the Python library for crawling the data is "ray", this example demonstrates the execution flow of a method for automatically maintaining version-controlled datasets based on a multi-agent agent system. The specific steps are as follows:

[0052] like Figure 1 As shown, step S1: Based on the list of legitimate Python libraries retrieved from the PyPI website, search for these libraries' projects on the GitHub website and obtain their metadata, including project stars, download address, latest version, and release date; filter out Python libraries with a total number of stars greater than 10,000 or a star increase of more than 1,000 in the past month. Combine this with the library version information contained in the local VersiCode dataset. For example, the ray library has more than 10,000 stars. Assuming the latest version of the ray library in the VersiCode dataset is 1.12.0, and newer versions 1.12.1 and 1.13.0 have been released, by comparing the latest version in the metadata with the library version records in the locally maintained dataset, determine that the data to be crawled is the source code packages of ray-1.12.1 and ray-1.13.0.

[0053] like Figure 2 As shown, step S2: Based on the list of Python libraries to be crawled obtained in step S1, a script is used to parse the `requires_python` field in the metadata of `ray-1.12.1` and `ray-1.13.0` using regular expressions to obtain the Python version environment required for the installed library version. Further, the `preprocess` module of the AexPy tool is used to sequentially crawl the source code packages of `ray-1.12.1` and `ray-1.13.0` that meet the corresponding Python versions. Further, the `extract` module of the AexPy tool is used to sequentially extract the APIs from the library versions, including modules, classes, functions, and attributes, forming the internal API knowledge base of each library version. Further, based on the extracted library version APIs, a script is used to compare consecutive library versions to find APIs that have evolved, forming an API evolution knowledge graph between library versions. The evolution types are referenced in the literature "How Do Python Framework APIs Evolve? An Exploratory Study".

[0054] like Figure 3 As shown, step S3: Based on the API evolution knowledge graph between library versions obtained in step S2, an LLM-based agent is used to extract code usage examples from the API docstring. For example, the "range_arrow" function in ray-1.12.1 is renamed to "range_table" in ray-1.13.0. The LLM agent is used to process their docstrings to extract code usage examples, as follows:

[0055] {"liabrary_version":"ray==1.12.1","API":"range_arrow",

[0056] "code_example":"import ray\n ds=ray.data.range_arrow(1000)\n ds.map(lambda r:{"v2":r["value"]*2}).show()"}

[0057] The code_example field is derived from the docstring of the range_arrow function. The extraction method involves writing the possible code example patterns in the docstring into a prompt, which serves as context to inform the LLM agent, thereby guiding it to identify and extract valid code snippets.

[0058] Next, an LLM-based agent is used to generate functional descriptions for the successfully extracted code snippets, for example:

[0059] {"liabrary_version":"ray==1.12.1","API":"range_arrow",

[0060] "code_example":"import ray\n ds=ray.data.range_arrow(1000)\n ds.map(lambda r:{"v2":r["value"]*2}).show()",

[0061] "functionality_description":"The code creates a dataset,maps eachelement to double its value,and displays the results."}

[0062] The `functionality_description` field is a functional description generated by the LLM agent for `code_example`. This functional description, along with relevant information such as the library version, is written into the prompt as part of the dataset. This provides context to the LLM agent, guiding it to generate code for the corresponding version and functionality. This yields metadata in tuple form, containing at least three core elements: library version, functional description, and code snippet, which is used for subsequent task-oriented processing.

[0063] like Figure 4 As shown, step S4: Based on the metadata obtained in step S3, perform dataset task processing. Taking the code completion task (token level) as an example, the following steps are taken: Replace "range_arrow" in the "code_example" field with " <mask>This means performing a masking process, resulting in "masked_code": "import ray\n ds=ray.data". <mask>(1000)\n ds.map(lambda r:{"v2":r["value"]*2}).show()”. This tells the LLM "library_verison", "functionality_description", and "masked_code" so that the LLM can perform inference and prediction. <mask>"Location," or version-sensitive area, is used to test the performance of LLM in version-sensitive code completion. Taking code migration tasks as an example, for evolution types involving cross-version changes, such as API renaming and relocation, based on the API evolution knowledge graph between library versions, each set of data involving two API versions is used as a test instance. The library version, feature description, and code snippet of ray==1.12.1 (source version), and the library version and feature description of ray==1.13.0 (target version) are provided to the LLM, allowing the LLM to infer that the target version contains the code snippet of the evolved "range_arrow" API. That is, the code migration task of the "range_arrow" API from ray==1.12.1 to ray==1.13.0 is implemented (in ray==1.13.0, range_arrow is renamed to range_table). At this point, the final form of the data is obtained, which is the updated data included in the VersiCode dataset.

[0064] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.< / mask> < / mask> < / mask> < / mask>

Claims

1. An automatic evolution method for version-controlled datasets based on large language models, characterized in that: It includes a data crawling module, a data parsing module, a data annotation module, and a task format processing module, specifically including the following steps: S1. The data crawling module collects metadata of Python library versions from multiple data sources through automated means to ensure the integrity and timeliness of the collected data, and performs preliminary screening of the data. S2, the data parsing module, uses automated methods to perform structured processing and preliminary cleaning of the crawled data, extracting key fields and standardizing data formats to ensure data consistency and accuracy, providing reliable input for subsequent annotation and processing; the specific steps of S2 are as follows: S201 optimizes the AexPy tool to meet the requirements of dataset construction, as follows: (1) Modify the internal code of AexPy to enable it to support multiple versions of Python environment; (2) By parsing the requires_python field of the library version metadata, the specified Python version is obtained and passed to the AexPy tool to alleviate the problem of incompatibility between library and dependency versions; (3) By modifying the internal code of AexPy, the missing dependencies can be automatically identified and installed using the predefined library-dependency mapping relationship; S202, based on Python library metadata, parses the requires_python field corresponding to each library version to obtain the Python version environment required to install the library version; S203 uses the preprocess function module of the AexPy tool to crawl the source code package of the corresponding Python version library version. S204 uses the extract function module of the AexPy tool to extract APIs and obtain the internal knowledge base of each library version. S205, based on the Python API evolution pattern, and based on the knowledge base of each library version, finds the APIs that have evolved between two consecutive library versions and obtains the API evolution knowledge graph between library versions. S3. The data annotation module automatically annotates the parsed data using LLM-based proxies and preset rules, and adjusts the annotation strategy according to different task scenarios. S4, the task format processing module, uses LLM-based agents and automated scripts to transform the labeled data into the format required for specific tasks, making it more suitable for the evaluation process and ensuring that the data meets the requirements of LLM performance evaluation scenarios. The specific steps of S4 are as follows: S401, based on metadata, is further processed into two task forms: code completion and code migration, which are contained in the VersiCode dataset; For code completion tasks, version-sensitive APIs are located based on AST parsing, and the target API is replaced with... <mask> The tags, combined with contextual hints including library version constraints, construct code completion data;< / mask> For code migration tasks, based on the knowledge graph of API evolution between library versions and combined with contextual hints containing library version constraints, code migration data pairs are constructed, including source version data and target version data.

2. The automatic evolution method for version-controlled datasets based on large language models according to claim 1, characterized in that: The specific steps for S1 are as follows: S101, Use a web crawler script to search the PyPI website and obtain a list of legitimate Python libraries; S102, retrieve Python library projects from the list on the Github website and obtain metadata, including the number of stars and release time of the project; S103, filtering target Python libraries based on Github star count threshold; S104, taking into account the local VersiCode dataset, only crawls the source code package of the new version of the existing library in the dataset, and crawls all historical versions within the specified time range for the new library; S105: Organize the relevant metadata of the two types of Python libraries and hand it over to the data parsing module for subsequent data processing.

3. The automatic evolution method for version-controlled datasets based on large language models according to claim 1, characterized in that: The specific steps for S3 are as follows: S301, using an LLM agent, provides an example of API extraction code usage based on the documentation in the metadata; S302 uses an LLM agent to generate functional descriptions for successfully extracted code snippets; S303 performs data cleaning based on heuristic rules, obtains successfully processed data, and records it as metadata, which includes at least three core elements: library version, function description, and code snippet.

Images