Method for evaluating and relieving third-party library API incompatibility problem in large model code completion

Abstract

The invention discloses a method for evaluating and relieving a third-party library API (Application Program Interface) incompatibility problem in large model code completion. The method comprises the following steps of: constructing a third-party library API knowledge base for a Python language third-party library; aiming at the problem that potential third-party library APIs in completion codes generated by a code completion system based on a large language model are incompatible in use, automation is carried out based on a third-party library API knowledge base; and a lightweight intervention mechanism is adopted to detect and repair an incompatibility problem in real time. According to the method, API change information is obtained through systematic analysis and version comparison of the third-party library APIs, incompatible use of the APIs is detected, and lightweight relief is achieved by dynamically injecting problem causes and API standard use information into cues; the API signature and real-time incompatibility detection service can be provided with extremely low performance overhead, and the development efficiency and the code reliability are practically improved.

Description

Technical Field

[0001] This invention relates to the technical field of incompatible API usage detection tasks in software engineering and program analysis, specifically to a method for evaluating and mitigating third-party library API incompatibility problems in large model code completion. Background Technology

[0002] Large Language Models (LLMs) drive a shift in code completion techniques from traditional statistical methods to a context-aware generative paradigm. These LLMs, through pre-training or fine-tuning on large-scale code corpora, possess the ability to generate code completion suggestions that are syntactically and semantically more relevant to identifiers, variables, methods, and even higher-level code architectures.

[0003] To improve development efficiency, modern software engineering practices commonly use Application Programming Interfaces (APIs) to call third-party library APIs. This dependency presents new challenges for code completion technology: the software ecosystem is inherently dynamic and evolving, and the rapid iteration of third-party library APIs leads to frequent additions, deprecations, or changes in API definitions. When the API recommended by the code completion tool is incompatible with the currently used library version, it may cause build failures or runtime behavior anomalies. This phenomenon is prevalent in rapidly evolving ecosystems with abundant library resources; we call this phenomenon Incompatible Third-party Library API Usage (ITAU). Figure 1 As shown, the `RandomForestClassifier` class in the scikit-learn library underwent a destructive change between versions 0.22 and 0.23: starting with version 0.23, a parameter named `cpp_alpha` was added to the constructor of this class, which did not exist in version 0.22. If the code completion model recommends this function without being aware of the actual library version used in the target environment, the generated code will trigger a `TypeError` exception and fail to run. This type of error caused by incompatibility between the code completion result generated by LLM and the API in the currently installed library version is a typical case of ITAU, which raises a key question: to what extent can an LLM-based code completion system adapt to the rapid evolution and version sensitivity of TPLs?

[0004] Currently, there is still a research gap regarding the ability of LLM-based code completion tools to handle ITAU issues in real-world development scenarios. While existing work has begun to focus on the sensitivity of LLM to library changes, it is often limited to static code generation tasks or controlled benchmarking scenarios. These datasets are typically built for specific library versions, requiring manual collection and annotation of samples. Although these methods can reveal to some extent whether large language models are sensitive to simple version-related signals (such as the introduction of deprecated APIs), they cannot provide a comprehensive automated solution for assessing ITAU issues caused by code completion tasks in real-world development environments, and fail to fully reflect the complexity and dynamism of actual development environments. Research has found that LLM frequently recommends deprecated Python APIs in code completion tasks, and struggles to consistently recommend the latest version of APIs. Similarly, studies have been conducted on the Rust and PyPI ecosystems, analyzing whether API calls generated by the model are compatible with specified package versions to evaluate LLM's ability to handle API evolution.

[0005] While these works provide valuable insights into the general limitations of LLMs in dealing with API evolution, they still have the following limitations when applied to real-world code completion scenarios: (1) Difficulty in providing precise dependency specifications for real-world development scenarios. Existing benchmarks directly provide all the dependency specifications required to generate target code for large language models, but this ideal condition is usually not met in real-world development scenarios. Developers typically only declare the range of dependency versions (e.g., pandas ≥ 1.3, < 2.0) or even omit version constraints entirely, leaving the package manager to dynamically resolve the specific dependency versions used during installation. This results in the actual dependency versions used changing due to the development environment and time, and dependency resolution being uncertain. Furthermore, developers are often unaware of the actual dependency versions used in their environment, especially when transitive dependencies or automated upgrade mechanisms exist. This lack of precise and stable dependency specifications presents a fundamental challenge: since existing research has not systematically evaluated the performance of LLM in situations where dependency information is ambiguous or missing, the performance of LLM-based code completion systems in such dynamic environments remains to be explored.

[0006] (2) Lack of fine-grained understanding of compatibility issues. Existing assessments of LLM code completion capabilities in API evolution scenarios mostly take a coarse-grained approach, focusing on high-level changes such as API additions, deletions, or deprecations. While these change categories can reflect the macro trends of API evolution, they neglect fine-grained destructive changes that occur more frequently in practice and are more likely to trigger runtime errors (such as AttributeError, TypeError, etc.). These subtle changes are particularly common in actively maintained libraries. When an LLM-based code completion tool recommends API usage that does not match the version of the installed library, it can easily lead to ITAU issues. However, existing benchmarks rarely cover such complex scenarios, leaving the ability of LLM to effectively identify and adapt to fine-grained API incompatibilities in real-world development environments unclear.

[0007] To overcome the above limitations, an empirical study was conducted to explore the ITAU problem of LLM-based code completion systems in real sound field environments: (1) To address limitation one, highly starred GitHub projects with complex dependency configurations were collected to simulate the scenario in a real development environment where library versions are not explicitly specified and are dynamically resolved by the package manager. (2) To address limitation two, the system analyzed fine-grained changes in API signatures across library versions to assess the actual compatibility issues caused by using third-party library APIs in a rapidly evolving environment.

[0008] Based on the above insights, this application proposes a method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for evaluating and mitigating the incompatibility problem of third-party library APIs in large model code completion.

[0010] The objective of this invention is achieved through the following technical solution: a method for assessing and mitigating API incompatibility issues in large model code completion, comprising the following steps: S1. Build a knowledge base for third-party library APIs for Python language third-party libraries; S2. To address potential incompatibility issues in third-party library API usage in the code completion code generated by the code completion system based on the large language model, an automatic evaluation is performed based on the third-party library API knowledge base. S3. For the incompatibility issues identified in the assessment, a lightweight intervention mechanism is used to detect and repair them in real time.

[0011] Furthermore, step S1 specifically includes the following sub-steps: S1.1 Define a unified API model to characterize the core structural information of third-party library APIs; S1.2 Select third-party libraries from PyPI and download their source code for each version to your local machine; create a global timeline based on the release timestamps of the third-party libraries, and standardize the version tags of the third-party libraries to obtain PyPI packages; S1.3. Based on the defined API model, extract API signatures from the source code of each version of the third-party library in the PyPI package. In the process of API signature extraction, a static import flow analysis mechanism is introduced to track the re-import, export and propagation of symbols between Python modules. That is, a directed graph is constructed with modules as nodes and import relationships as edges, and the reachable alias path set of each API is calculated by passing closures. S1.4 Perform cross-version aggregation and difference analysis on each third-party library to detect structural changes, build a version index based on API signatures, and record changes that occur between adjacent versions, including add operations, remove operations, and modify operations, in order to build a third-party library API knowledge base.

[0012] Furthermore, the API model is a four-tuple model, including name, formal parameters, return type, and scope.

[0013] Furthermore, the third-party API knowledge base includes a versioned API signature library for recording all API signatures in each version and an inter-version API difference library for capturing changes between adjacent versions.

[0014] Furthermore, step S2 specifically includes the following sub-steps: S2.1 Collect Python projects that use third-party libraries from GitHub as research corpus, and filter them based on project information and dependency coverage as the final raw data; S2.2 Extract potential third-party library API incompatibility issues from the completion code in Python projects through three stages: third-party library API usage detection, dependency declaration verification and API stability check, and API usage isolation. S2.3 For potential third-party library API incompatibility issues in the completed code, construct the code completion task in the local context of the API call in the file where the third-party library is located. Specifically, the code completion task is constructed through two levels: API-level completion and function-level completion. S2.4 Construct prompt words based on the code completion task, and use a large language model to generate completed Python code based on the prompt words; S2.5 For the completed Python code, extract the API signatures of its calls, then compare them with the expected API signatures of the target version of the third-party library, and use two API signature-level compatibility evaluation metrics, namely compatibility rate and destructive completion rate, to evaluate the compatibility perception capability of the large language model in the real third-party library evolution scenario.

[0015] Furthermore, the third-party library API usage detection specifically includes: performing static analysis on the completion code in the Python project to identify all API calls in the completion code, and standardizing each API call into an FQN containing the full path, module, class name, and API name; identifying the third-party library API calls by matching the FQN with the API signature in the third-party library API knowledge base; The dependency declaration verification and API stability check specifically include: first, obtaining the version range corresponding to each third-party library from the dependency declaration of the Python project; then, extracting the version information of each API call in the third-party library API call and comparing it with all API signatures of the corresponding version range in the third-party library API knowledge base to verify whether the third-party library API call is compatible with the version range specified in the dependency declaration, and identifying third-party library API calls that may cause incompatibility issues in the use of third-party library APIs during the library version evolution process; The API usage isolation specifically includes: constructing a call graph for third-party library API calls obtained after dependency declaration verification and API stability checks; capturing the call chain within the file by traversing the call graph of the current file to isolate the third-party library API calls to the scope of the current file, thereby obtaining the final potential third-party library API usage incompatibility issues.

[0016] Furthermore, the formula for calculating the compatibility rate is: In the formula, CR represents the compatibility rate. This represents the set of prompts being evaluated. Large Language Model (LLM) targets prompt words The generated completed Python code, This represents the set of completed Python code that is compatible with the API signature of the target version of third-party libraries. This is an indicator function that returns 1 if the condition is true, and 0 otherwise. The formula for calculating the destructive completion rate is as follows: In the formula, Indicates the destructive completion rate. This represents the set of completion results where the completed Python code violates the corresponding API signature.

[0017] Furthermore, in step S3, the lightweight intervention mechanism specifically includes the following process: Real-time detection: Continuously monitor third-party library API calls generated during code completion. When the code completion system based on the large language model identifies a third-party library API call, it immediately queries the third-party library API knowledge base to obtain the API signature for the current version. Then, it compares the identified third-party library API call with the queried API signature, and marks the third-party library API call as a potential incompatibility issue if any destructive rule is violated. At the same time, regardless of whether a potential incompatibility issue is detected, the code completion system based on the large language model will present the queried API signature to the developer in real time as a reference to help them immediately determine the incompatibility issues of the third-party library API call. Lightweight Repair: For real-time detection results, a lightweight repair mechanism is used to fix potential incompatibility issues in the markings.

[0018] Furthermore, the lightweight repair mechanism specifically includes: For each potential incompatibility issue, namely incompatible third-party library API calls, two types of key information are injected into the original prompt words to obtain new prompt words. The new prompt words are then used to guide the large language model to generate compatible third-party library API calls, thereby completing the repair of potential incompatibility issues. The two key types of information include: the type and cause of potential incompatibility issues in the current version of the third-party library, and the API signature of a specific version extracted from the API knowledge base of the third-party library.

[0019] Compared with existing technologies, the beneficial effects of this invention are as follows: The main technology of this invention is program analysis and prompting engineering. It systematically analyzes Python third-party library APIs, compares versions to obtain API change information, detects API incompatibility, and achieves lightweight mitigation by dynamically injecting problem causes and API specification usage information into prompts. This invention constructs a third-party library API knowledge base based on static analysis technology and uses example code parsing and evaluation of completion results to effectively assess the ability of a large language model to cope with fine-grained API changes. This invention also provides a lightweight mitigation method for fine-grained incompatibility issues, which can provide practical API signature information and real-time incompatibility detection services with minimal performance overhead, effectively improving development efficiency and code reliability; it is more suitable for use in actual development scenarios. Attached Figure Description

[0020] Figure 1This is an example of an ITAU issue caused by the evolution of third-party libraries; Figure 2 This is an architectural flowchart of the assessment and mitigation methods for third-party library API incompatibility issues in large model code completion; Figure 3 This is an example of transitive import in the API signature extraction process; where, Figure 3 (a) in the text is an import statement; Figure 3 (b) in the table of contents is the directory tree; Figure 4 This is an example diagram of Algorithm 1 for locating potential API incompatibility issues with third-party libraries; Figure 5 This is a diagram illustrating the construction of code completion suggestions. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0024] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0025] This invention proposes an evaluation and mitigation method for third-party library API incompatibility issues in large model code completion, referred to as the VerComFix method, based on... Figure 2The example shown is a novel automated assessment and mitigation framework called VerComFix, designed to systematically evaluate ITAU issues caused by LLM-based code completion systems in real-world development environments. While the system is primarily focused on the Python ecosystem, its architecture is language-independent, allowing seamless migration to other programming language ecosystems with abundant library resources. This assessment and mitigation framework first collects several historical versions of widely used third-party libraries and extracts API signatures based on Abstract Syntax Tree (AST) analysis, thereby constructing a versioned third-party library API knowledge base. This knowledge base reflects the dynamic evolution trajectory of APIs in real-world projects based on fine-grained API evolution patterns (such as parameter additions / deletions, default value changes, and return value type adjustments), and supports the systematic identification of potentially compatibility-breaking uses. The knowledge base covers 129,970 versions of 2,178 libraries, containing a total of 496,369,872 unique API signatures. Subsequently, it mines highly-starred and actively maintained open-source repositories on GitHub to locate API call points of third-party libraries that may be affected by API changes. By masking the target API calls while preserving the context to construct prompts that closely resemble real-world development scenarios, these prompts are transformed into code completion tasks. Ultimately, 10,867 code completion tasks were constructed, and evaluations were conducted based on six state-of-the-art LLM code completion systems. Unlike traditional evaluation methods that use surface text similarity metrics, two novel evaluation metrics were designed for this code completion task: CompatibilityRate (CR) and Breaking Completion Rate (BCR), to measure whether the API calls in the completion results match the actual library versions used. Furthermore, the impact of prompt configuration and API evolution types on the model's ability to avoid ITAU was further analyzed.

[0026] See Figure 2 The method specifically includes the following steps: S1. Build a knowledge base for third-party library APIs for Python.

[0027] It should be noted that, to achieve version-aware compatibility analysis, a fine-grained third-party library API knowledge base was built to capture the cross-version evolution trajectory of third-party library APIs. The following section details the process of obtaining multi-version packages from libraries from PyPI, extracting structured API signatures, and organizing them into a knowledge base that supports downstream code completion tasks for evaluation. PyPI is the official and primary third-party package repository for the Python language.

[0028] S1.1 API Model Definition: Define a unified API model to characterize the core structural information of third-party library APIs, supporting fine-grained compatibility analysis across library versions. This API model supports systematic detection of destructive changes as described in Table 1 (such as changes to API name, parameter types, return value types, etc.). The API model is a four-tuple model, including Name, Param, Return type, and Scope, and can be represented as: In the formula, This section defines the API model. `Name` represents the fully qualified name (FQN) of the API, including the module path and class context, such as "sklearn.ensemble.RandomForestClassifier.init". `Param` is an ordered list of parameter names, such as "n_estimators" and "ccp_alpha", which includes parameters with default values. `Return` represents the type of the return value or a statically inferred symbolic return expression, such as the return type of "numpy.mean", which may be float or ndarray depending on the input. Although Python is a dynamically typed language, its return type can usually be inferred from type annotations or docstrings. `Scope` distinguishes between top-level methods (such as "math.sqrt"), class methods (such as "pandas.DataFrame,head"), and class attributes (such as "sklean.SVC.support_"). This `Scope` field is crucial for detecting changes in inheritance structure, visibility adjustments, and attribute semantic shifts. By explicitly tracing the above field elements across versions, a systematic API difference analysis can be achieved, which helps to identify potential compatibility issues in LLM-generated code completion results under real third-party library evolution environments.

[0029] Table 1: Fine-grained API evolution patterns, matching rules, and compatibility symptoms Explanation of symbols in Table 1: These represent classes, functions, properties, and parameters, respectively. These represent the sets corresponding to the original versions. This represents the set corresponding to the new version; Representation Class The base class; Representation function The return type; This indicates the complete parameter signature.

[0030] S1.2 PyPI Package Analysis: To support compatibility analysis in a real development environment, widely used third-party libraries were selected from PyPI, focusing on the top 2500 mainstream third-party libraries in terms of download volume or dependency count, covering multiple key areas such as data science, web development, and machine learning. To ensure rapid and reliable access, a local PyPI mirror was built and maintained. This means that the source code of each version of the third-party libraries to be analyzed is downloaded locally, avoiding the need to download them from the external network each time, reducing over-reliance on the central repository, and periodically synchronizing with upstream repositories to achieve incremental updates. This design supports incremental updates and can effectively track the evolution history of libraries without over-reliance on the central repository. Because the Python community's enforcement of version number specifications is relatively loose, there are inconsistencies in pre-release tags and version number increments. Therefore, a global timeline was established based on the official release timestamps to standardize the version tags of each third-party library, ensuring that the version tag format of all third-party libraries remains consistent. This strategy ensures consistency when comparing adjacent versions and detecting API level changes. For example, the original version tags of third-party library A and third-party library B are A-0.1A-0.5-pre A1.0 A-2.0 and B-0.0.1 B-0.0.2 B-1.0.2, respectively. After standardizing the version tags of the third-party libraries, we get: A-0.1 A-0.2 A-1.0 A-2.0, B-0.1 B-0.2 B-1.0.

[0031] S1.3 API Signature Extraction: Based on the defined API model, extract API signatures from the source code of each version of third-party libraries in the PyPI package. In the API signature extraction process, a static import flow analysis mechanism is introduced to track the re-import, export and propagation of symbols between Python modules. That is, a directed graph is constructed with modules as nodes and import relationships as edges, and the reachable alias path set of each API is calculated by passing closures.

[0032] In this embodiment, to construct a precisely structured, versioned API knowledge base, fine-grained API signatures are extracted from the source code of various versions of third-party libraries based on a defined API model. This process first statically analyzes all Python source files within the PyPI package. Then, it constructs an abstract syntax tree for each Python module and traverses the tree structure to identify the "FunctionDef" (i.e., ordinary function definition) node and "AsyncFunctionDef" (i.e., asynchronous function definition) node corresponding to the API entry point, as well as the related "ClassDef" (i.e., class definition) context. For each identified API, information from the formally defined API model is extracted: the FQN composed of module paths and class inheritance hierarchy, an ordered parameter list containing positional and keyword arguments, return type, and scope (such as top-level functions, class methods, class attributes, etc.).

[0033] However, Python's flexible dynamic mechanism allows the same API to be accessed through multiple alias paths, posing challenges to path resolution. Due to Python modules' transitive imports and symbol re-exports, an API defined deep within a package (such as "pandas.io.excel._base.read_excel") might be invoked through a more concise, user-friendly path (such as "pandas.read_excel"). Therefore, to construct a complete set of valid access paths consisting of all aliases, a static import flow analysis mechanism is introduced during API signature extraction to track symbol re-imports, exports, and propagation between modules. Specifically, a directed graph is constructed based on all "import" and "from-import" statements, with modules as nodes and import relationships as edges, and the set of reachable alias paths for each API is calculated using transitive closures. This static import flow analysis mechanism enables the simultaneous resolution of the API's original definition location and all valid access paths within a given library version.

[0034] For example, such as Figure 3 As shown in (a), the API "read_excel" was initially defined in the "pandas.io.excel._base" module via a standard function declaration. However, through the statement "from excel._base import read_excel", this function was re-exported in "pandas / io / api.py", and finally exposed in "pandas / init.py" in the package root directory through the statement "from pandas.io.api import read_excel". Figure 3As shown in (b) above. Therefore, although the actual definition of the function is located deep within an inner module, it can still be accessed via the simplified path "pandas.read_excel". The static import stream analysis mechanism captures such transitive relationships by constructing the following symbolic stream chain: By calculating the transitive closure of such import chains, it can be further deduced that: By integrating import flow information, a complete set of access paths is built for each API in the versioned third-party library API knowledge base. Each API entry includes a canonical signature extracted from its source definition location, as well as a set of alias paths identified through import propagation analysis. This allows the knowledge base to accurately reflect how developers actually access APIs and supports comprehensive detection of disruptive changes occurring in the definition location and exposed access paths.

[0035] S1.4 API Knowledge Base Construction: After extracting API signatures for each version, cross-version aggregation and difference analysis are performed on each third-party library to detect structural changes. A version index is built based on the API name and detailed signature, i.e., the API signature, and fine-grained changes that occur between adjacent versions are accurately recorded, including addition operations, removal operations, modification operations, etc., in order to build a third-party library API knowledge base.

[0036] The third-party API knowledge base includes a versioned API signature library for recording all API signatures in each version and an inter-version API difference library for capturing fine-grained changes between adjacent versions.

[0037] The current knowledge base covers 2178 software packages, containing 496,369,872 unique API signatures from all versions. While the initial target was the 2500 third-party libraries with the highest download volume or dependencies, some libraries were ultimately excluded for the following reasons: some packages lacked analyzable APIs (such as plain data packages), others lacked historical versions on PyPI, and still others failed to build due to dependency or compilation issues. This knowledge base forms the basis for conducting compatibility assessments, enabling precise tracing of the evolution of specific APIs and further determining the effectiveness of API usage under specific library versions.

[0038] S2. To address potential incompatibility issues in the third-party library APIs used in the code completion code generated by the code completion system based on the large language model, an automatic evaluation is performed based on the third-party library API knowledge base.

[0039] S2.1 Raw Data Collection: To simulate the completion behavior of an LLM-based code completion system in a real-world development scenario, a batch of actively maintained Python projects that widely use third-party libraries were selected from GitHub as the research corpus. The raw data collection process included two key steps: (1) Project Selection: Non-forked projects created between June 1, 2024 and December 31, 2024, with data crawling to begin on June 1, 2025. This time window ensures sufficient data collection while guaranteeing that the collected data does not overlap with the training corpus used by most large code models published before June 1, 2024. To ensure project quality and activity, only projects with at least 50 stars, at least 10 forks, and updates within the past 12 months are retained. To reliably resolve dependency version constraints, projects are further required to include explicit dependency declaration files (i.e., requirements.txt or setup.py).

[0040] (2) Dependency Coverage Screening: Projects with rich dependency graphs are more likely to involve complex API usage patterns and face greater compatibility risks due to version-specific changes. Minimalist or single-purpose projects with fewer than 5 declared dependencies were excluded to ensure sufficient interaction between the project and third-party libraries.

[0041] S2.2 Locating potential compatibility issues: see as follows Figure 4 Algorithm 1 shown is designed to identify Python projects. The code completion information may be generated by an LLM-based code completion system and third-party libraries. Specific version Potential ITAU incompatibility issues. Due to the rapid evolution of third-party libraries, their APIs often change between different versions; however, LLM-based code completion systems, lacking version awareness, are prone to generating code that references outdated API usage, leading to risks of incompatibility with the current library version. Specifically, potential third-party library API incompatibility issues in the completed code of Python projects are extracted through three stages: third-party library API usage detection, dependency declaration verification and API stability checks, and API usage isolation.

[0042] (1) Third-party library API usage detection: Static analysis is performed on the completion code in the Python project to identify all API calls in the completion code, and each API call is standardized into an FQN containing the full path, module, class name and API name; the third-party library API calls are identified by matching the FQN with the API signature in the third-party library API knowledge base.

[0043] This step aims to extract the Python project. The code completes all third-party library API calls present in the project. Specifically, it completes the code for the project. Perform static analysis to identify all API calls that appear in FQN (such as "img.img_to_graph") or its abbreviation (such as "img_to_graph"). Figure 4 Line 5 in [the document / reference]. To ensure compatibility with third-party library API knowledge bases, all API calls are standardized to the "ABCAPI_Name" format FQN (which includes the full path, module, class name, and API name). Figure 4 (Line 7 in the text).

[0044] To resolve API calls to FQNs, all Python source files in the project are first traversed to identify all function and method calls. Then, the AST is traversed to find the corresponding method call or API reference. For example object method calls, object type resolution is used (…). Figure 4 Lines 8-9 in the code ensure that methods are fully qualified by both the class and module names. For example, assuming "dt" is an instance of type "pandas.DataFrame", it will be guaranteed that calling the method "loc()" on the object "dt" (i.e., "dt.loc()") will resolve to "pandas.DataFrame.loc()". The use of package, class, and function aliases presents another challenge to FQN resolution. Figure 4 Lines 10-11 of the code demonstrate how Python's "import-as" feature allows developers to create aliases for packages or functions (e.g., "import pandas aspd", "from pandas import DataFrame"), necessitating that API calls using aliases (e.g., "pd.DataFrame.loc()") be mapped back to their original paths (e.g., "pandas.DataFrame.loc()"). This mapping from aliases to original paths is constructed by resolving "import" and "from-import" to achieve reverse substitution at API call points. For example, when "pandas" is imported with the alias "dt", "pd.DataFrame.loc()" is mapped back to "pandas.DataFrame.loc()". Similarly, when a developer uses "from torch.linalg import lstsq", "lstsq()" is mapped back to "torch.linalg.lstsq()".

[0045] After completing the lightweight static analysis, the standardized FQN corresponding to each API call will be obtained. By matching it with the API signature in the third-party library API knowledge base, the API calls of the third-party library to be analyzed will be identified. Figure 4 (Lines 12-13 in the text).

[0046] (2) Dependency declaration verification and API stability check: First, obtain the version range corresponding to each third-party library from the dependency declaration of the Python project. Then, extract the version information of each API call in the API call of the third-party library and compare it with all API signatures of the corresponding version range in the API knowledge base of the third-party library to verify whether the API call of the third-party library is compatible with the version range specified in the dependency declaration and identify the API calls of the third-party library that may cause incompatibility issues in the use of the third-party library API during the evolution of library version.

[0047] This step aims to verify whether the extracted API calls to be identified from third-party libraries are compatible with the dependency version range declared in the project, and to identify APIs that have undergone significant changes in library version evolution and are more likely to cause potential ITAU issues during code completion.

[0048] First, let's look at the project dependency declaration. Get each third-party library Corresponding version range ( Figure 4 Line 16 in the document defines the expected range of library versions for the project to function correctly. Subsequently, third-party library API calls are extracted. Each identified API call point The version information is compared with the corresponding version range in the third-party library API knowledge base. Compare all API signatures ( Figure 4 (Lines 17-18 in the code). This operation ensures that the target API uses the version range specified in the project dependency declaration. It is compatible, and there is no possibility of incompatible changes. Figure 4 Line 19 in [the document / reference]. For example, a certain inventory is constrained by a version range (e.g., between version [number]). With version If the API signature remains stable within a specified version range (between [the specified range]), then no changes that could lead to incompatibility should have occurred. For dependencies with fixed version constraints and range version constraints, the API signature within the project dependency declaration constraint range is analyzed to ensure no changes have occurred. For unconstrained dependencies, the default installation method of package managers such as pip is simulated, mapping the latest available version to the project's commit timestamp and then verifying it according to the former's verification process. Furthermore, since within the specified version range... APIs that undergo significant changes are more likely to cause potential ITAU issues; therefore, the use of these third-party library APIs will be prioritized. Figure 4 (Lines 20-21 in the text).

[0049] (3) API usage isolation: For the API calls of third-party libraries obtained after dependency declaration verification and API stability check, a call graph is constructed; by traversing the call graph of the current file, the call chain within the file is captured, so as to isolate the API calls of third-party libraries to the scope of the current file, and thus obtain the final potential API usage incompatibility issues of third-party libraries.

[0050] To focus on the analysis objectives and eliminate irrelevant code interference, a lightweight call graph was constructed around the API call points of third-party libraries obtained after dependency declaration verification and API stability checks. This graph does not involve complete call chain analysis across the entire project or modules; it only captures the call chain within the current file by traversing the current file's AST. Figure 4 Line 24 in [the document / section]. This step aims to isolate third-party library API usage to the current file scope, thereby eliminating noise introduced by cross-file dependencies or external module calls. Figure 4 (Lines 25-26 in the code). This process helps prevent repository-level code structure from interfering with completion behavior, ensuring that subsequent analysis focuses on the file-local direct context where the API is used.

[0051] S2.3 Constructing Code Completion Tasks: For potential third-party library API calls with incompatibility issues in the code completion, construct code completion tasks within the local context of the API call in the file where it is located. Specifically, the code completion tasks are constructed through two levels: API-level completion and function-level completion, in order to evaluate the adaptability of the large language model to different generation granularities.

[0052] In typical code completion scenarios, developers usually pause after manually writing a few lines of code, relying on LLM to generate subsequent code based on the existing context. Inspired by this actual development process, a code completion task is constructed around the API call points of third-party libraries that may cause ITAU issues, extracting the local context preceding that call point in the file. To evaluate LLM's adaptability to different generation granularities, two different levels of code completion tasks were designed: (1) API-level completion: Masking the lines of code containing API calls to the target third-party library, prompting the large model to only complete the lines.

[0053] (2) Function-level completion: Provides a function header containing the function name and parameter list, and prompts the large model to complete the function body. This task can better reflect the real development scenario where developers rely on LLM to guess the deep intent and automatically generate complete function logic.

[0054] like Figure 5 As shown, it also provides explicit dependency configuration information for large models (such as "scikit-learn"). 1.1.3” and the code snippet to be completed, the large model needs to generate code completion results that are compatible with the specified dependency versions. A total of 10,867 code completion tasks were constructed for evaluation.

[0055] S2.4 Code completion based on a large language model: Based on the code completion task constructed in step S2.4, prompt words are constructed, and the large language model is used to generate the completed Python code based on the prompt words.

[0056] To comprehensively evaluate the performance of current mainstream LLMs in code completion tasks, several LLMs were selected as shown in Table 2, including five state-of-the-art open-source models: CodeGen, StarCoder, CodeLlama, Deepseek-Coder, and Deepseek, as well as one closed-source model, GPT-4o. The selected open-source models cover three typical training paradigms: base models pre-trained on general corpora, code models fine-tuned based on base models, and dedicated models trained entirely on code data. For the open-source models, a local API server based on vLLM was deployed to provide a unified interface for LLM services and inference. For the closed-source GPT-4o, calls were made through the official online API. The training data cutoff date for all models used in this evaluation is earlier than June 1, 2024, and there is no overlap with the selected data.

[0057] Table 2: Large Language Models Used for Evaluation In the code completion task, the large model needs to generate the next line of Python code following specific input provided by prompts. To ensure output stability, all open-source models employ a greedy decoding strategy, selecting the token with the highest probability at each step; the temperature parameter of GPT-4o is explicitly set to 0 to maintain consistency with the other models. The maximum generation length for all models is uniformly set to 100 tokens.

[0058] S2.5. Completion Result Labeling and Evaluation: For the completed Python code, extract the API signatures of its calls, then compare them with the expected API signatures of the target version of the third-party library, and use two API signature-level compatibility evaluation metrics, namely compatibility rate and destructive completion rate, to evaluate the compatibility perception capability of the large language model in the real third-party library evolution scenario.

[0059] This step aims to determine whether the APIs called in the LLM completion results are compatible with the target version of the third-party library. To this end, the standardized FQN of the API calls is first extracted using the same parsing strategy as in step S2.2. Furthermore, the AST nodes corresponding to the API calls from the third-party library are analyzed to extract the list of arguments passed to the API, associated values, and related return value usage. This information is the core basis for evaluating the structural correctness of the API calls and determining whether they are consistent with the expected API signature under the declared version, forming the foundation for subsequent compatibility analysis.

[0060] To evaluate the compatibility awareness capability of LLM in real-world third-party library evolution scenarios, two compatibility-oriented evaluation metrics are proposed: Compatible Rate (CR) and Breaking Completion Rate (BCR). Unlike traditional metrics based on text similarity, CR and BCR focus on API signature-level compatibility rather than surface similarity.

[0061] (1) Compatibility Rate: The CR metric is used to measure the proportion of valid completion results that match the actual API signature in the target version of the third-party library among all generated completion results. Due to the inherent flexibility of API calls (such as optional parameters), five matching rules as shown in Table 3 are designed to determine whether a specific API call is compatible with the corresponding API signature.

[0062] In the formula, CR represents the compatibility rate; This represents the set of prompts being evaluated; It is an indicator function that returns 1 if the condition is true, and 0 otherwise. Indicates LLM for prompt words The generated, completed Python code; This represents a set of completed Python code that is compatible with the API signature of a specific version of a third-party library.

[0063] (2) Destructive Completion Rate: The BCR metric is used to quantify the proportion of completion results containing ITAU issues. These completion results will inevitably lead to serious incompatibility issues (such as runtime or compilation failures) under the current third-party library version and should be considered as serious defects. Four destructive usage matching rules as shown in Table 3 are designed to determine whether the current completion result should be marked as "destructive completion".

[0064] In the formula, Indicates the destructive completion rate. This represents the set of completion results where the completed Python code violates the corresponding API signature.

[0065] Unlike other similarity metrics that only provide coarse-grained output fidelity assessments (such as "Exact Match"), CR and BCR reveal the ability of a large language model to perceive library version evolution from a fine-grained perspective. An ideal code completion model should exhibit high CR values ​​and low BCR values. To ensure the reliability of the evaluation results, completion results marked as "uncertain" or "other" (such as calls to unresolved function signatures) are actively excluded when calculating BCR.

[0066] Table 3: API Compatibility and Destructive Usage Rules It should be noted that this step aims to determine whether the APIs called in the LLM completion result are compatible with the target version of the third-party library. The compatibility judgment is based on rules C1-C5 in Table 3 (all five conditions must be met simultaneously). Only when rules C1-C5 in Table 3 are met simultaneously is it considered "compatible". Violating one or more of C1-C5, and meeting any of rules B1-B4, is considered "destructive completion". The compatibility rate (CR) and destructive completion rate (BCR) are only used to assess the probability of incompatibility issues in the LLM completion process.

[0067] For example, a systematic empirical analysis of the automatic evaluation method for the proposed incompatibility problem is conducted around the following three core research questions: (1) Model perspective: To what extent will the most advanced LLM-based code completion system cause ITAU problems?

[0068] We measure the CR and BCR metrics of the completion results of several mainstream LLMs in a real open-source project environment under the constraint of third-party library versions, and evaluate the compatibility performance of different models.

[0069] We used two metrics, CR (higher is better) and BCR (lower is better), to compare the version compatibility of different LLMs at two granularities: API-level completion and function-level completion. Table 4 comprehensively shows the compatibility performance of current mainstream LLMs when handling code completion at different granularities. Overall, the compatibility performance of current mainstream LLMs in code completion still has significant bottlenecks. In API-level completion tasks, the CR of all models is below 32%, while the BCR remains high (between 45.99% and 68.50%). The situation is even more severe in function-level completion tasks, where the CR of all models drops below 7%, while the BCR often exceeds 80%. These results indicate that although LLM-based code completion systems have made some progress recently, generating completion results compatible with specific versions of third-party libraries remains a major challenge.

[0070] Of all the evaluated models, GPT-4o achieved the best performance: a CR of 31.78% and a BCR of 45.99% for API-level completion tasks; and a CR of 5.91% and a BCR of 73.35% for function-level completion tasks. While still far from ideal, these results indicate that GPT-4o is more capable of generating version-consistent completion results. In contrast, open-source models such as CodeGen and StarCoder lag significantly behind: their CRs at the API level are as low as 5.86% and 12.02%, respectively, and further decrease to 1.04% and 5.38% at the function level. CodeLlama and DeepSeek-Coder exhibit moderate performance (CR of approximately 28%) on API-level tasks, but show a significant drop in performance on function-level tasks.

[0071] Furthermore, these results also indicate that function-level completion tasks are more challenging than API-level ones: the CR (Complete Rate) of all models drops sharply while the BCR (Browser Completion Rate) rises significantly. This performance degradation stems from the significantly increased difficulty in generating longer, context-dependent code snippets. In this scenario, models are more likely to ignore version-specific constraints, leading to incompatible code completion outputs.

[0072] This reveals that current large language models still lack sufficient ability to perceive compatibility in code completion tasks, and even the most advanced models have significant room for improvement.

[0073] Table 4: RQ1: Comparison of CR and BCR of different models in API-level and function-level completion tasks. (2) Tip word perspective: How different types of dependency version declarations will affect the model’s ability to avoid ITAU.

[0074] Four prompt word configuration modes were designed: omission (no dependency information provided), fixed version, range constraint, and no constraint, to evaluate the impact of different dependency version information provision methods on model compatibility awareness and the tendency to generate incompatible APIs.

[0075] The impact of using different dependency configurations for prompts on LLM compatibility awareness was evaluated. As shown in Table 5, providing more precise dependency information significantly improves model performance. For example, GPT-4o achieved the highest CR (32.99%) and BCR (50.34%) with a pinned configuration; when dependency information was omitted, its CR was only 23.24%, while its BCR reached 66.25%. CodeLlama and DeepSeek-Coder showed similar trends: compared to omitting dependency information in the prompts, using pinned prompts reduced the BCR by more than 10 percentage points. Explicitly specifying dependency constraints helps the model avoid ITAU, ensuring that the generated completion results are consistent with specific API versions. Conversely, the models performed worst when dependency information was omitted: CodeGen and StarCoder both had CRs below 10%, while several models had BCRs exceeding 66%. This indicates that in the absence of dependency prompts, the model will mainly rely on prior knowledge learned during the pre-training stage for completion, often generating outdated or incompatible API calls. Each model exhibits moderate performance under both constrained and unconstrained configurations: while showing some improvement over complete omission, it still falls short of the performance of fixed-version cue words. For example, GPT-4o achieves a CR of 31.49% with constrained cue words, but its BCR is still higher than that of the fixed-version configuration. This indicates that broad or incomplete version descriptions introduce ambiguity: although the model can utilize some contextual cues, it struggles to consistently and stably resolve the fine-grained evolution of the API.

[0076] This reveals that dependency configuration at the prompt word level has a significant impact on ITAU occurrence rates. For all LLMs, using a fixed version configuration has the strongest ITAU suppression effect, while omitting dependency information configuration will cause more ITAU issues.

[0077] Table 5: RQ2: The impact of different prompt word configurations on the compatibility of completion results (3) API evolution pattern perspective: How different API evolution types will affect the probability of ITAU occurrence in LLM generation completion.

[0078] This paper analyzes the impact of three different API evolution patterns—API name changes, parameter adjustments, and return type changes—on the frequency of incompatible completion.

[0079] Table 6 illustrates the impact of different API evolution patterns on the probability of ITAU occurrence. Experimental results show that fine-grained, especially disruptive API changes involving parameter and return value types, are the main cause of compatibility issues in all LLMs, with an impact comparable to that of API name changes.

[0080] API name changes resulted in the lowest completion rates (CR) for most models, such as CodeGen at only 2.46% and StarCoder at 11.89%; however, this was accompanied by extremely high bounded completion rates (BCR), such as StarCoder at 76.33% and CodeGen at a staggering 80.65%. This is because the models tend to reproduce old identifiers memorized during pre-training rather than adapting to newly introduced names. Once the identifiers change, these fixed memory associations are broken, resulting in incompatible API calls. It is noteworthy that parameter changes and return value type changes also pose significant challenges to LLM code completion. For parameter changes, models with unchanged CRs exhibited extremely low CRs (CodeGen at only 3.57%, StarCoder at 10.00%, and GPT-4o at 24.81%), while BCRs often exceeded 60%. Return value type changes showed a similar trend: although CRs improved slightly (e.g., CodeLlama at 32.74% and StarCoder at 30.97%), BCRs remained high, with GPT-4o reaching as high as 72.45%.

[0081] The above results highlight the importance of focusing on studying the fine-grained, disruptive changes that occur during API evolution. The evaluation results related to parameter and return value type changes reveal the fundamental limitations of current LLMs in capturing subtle semantic changes: although the generated code completion appears syntactically correct, the consistently high BCR values ​​indicate that they frequently generate incompatible code in dynamically evolving API environments. Since parameter lists and return values ​​directly carry functional semantics, even extremely minor changes can lead to incompatibility, making these types of errors particularly alarming. These subtle yet far-reaching changes are frequent in the modern software ecosystem, exposing the vulnerability of current LLM code completion systems in handling real-world engineering scenarios.

[0082] This reveals that disruptive changes to fine-grained APIs significantly exacerbate ITAU risks, indicating that LLMs struggle to effectively handle semantic-level API evolution patterns that go beyond simple identifier mappings.

[0083] Table 6: RQ3: Impact of different API evolution modes on the compatibility of completion results S3. For the incompatibility issues identified in the assessment, a lightweight intervention mechanism is used to detect and repair them in real time.

[0084] It should be noted that due to frequent and continuous changes in software libraries and fine-grained APIs, retraining the LLM using the latest corpus to address the ITAU problem is not feasible. Therefore, focusing on intervention at the inference stage of the large language model and at the prompt word level, and detecting and correcting incompatibility issues in real time without modifying the underlying large language model, is a more practical and effective solution. Based on the aforementioned research findings, this step designs two lightweight intervention mechanisms aimed at alleviating the ITAU problem in LLM code completion systems, the main steps of which are as follows: S3.1 Real-time Detection: This step aims to continuously monitor third-party library API calls generated during code completion. When the LLM-based code completion system identifies a third-party library API call, it immediately queries the third-party library API knowledge base to obtain the API specification signature for the current version. This query process is extremely efficient, with most request response times below 0.03 milliseconds, and the performance overhead introduced by interactive code completion is negligible. Subsequently, the LLM-based code completion system compares and verifies the generated third-party library API call with the API specification signature according to the compatibility rules defined in Table 3. If any destructive rule is violated, the call is marked as a "potential incompatibility issue" and recorded for subsequent repair. Simultaneously, regardless of whether a potential incompatibility issue is detected, the LLM-based code completion system will present the obtained API specification signature to the developer in real time as a reference, assisting them in immediately judging the correctness of the third-party library API call. This dual-protection mechanism not only achieves real-time compatibility verification but also provides "just-in-time guidance" from developers, effectively preventing ITAU issues from remaining and ultimately entering the codebase.

[0085] S3.2 Lightweight Repair: A lightweight repair mechanism is proposed based on real-time detection results to fix potential incompatibility issues marked in the real-time detection results. This mechanism guides the large language model to generate compatible third-party library API calls through prompt words, without retraining the large language model. For each detected incompatible third-party library API call, the mechanism dynamically injects the following two types of key information into the original prompt words: ① the type and cause of the potential incompatibility issue in the current version of the third-party library; ② the standard API signature of the specific version extracted from the third-party library API knowledge base. Subsequently, the mechanism guides the large language model to regenerate only the minimum code snippet surrounding the third-party library API call while maintaining the context structure. This signature injection-based repair strategy has extremely low runtime overhead and is applicable to any LLM that supports natural language and code completion.

[0086] For example, the following experimental procedure was used to evaluate the effectiveness of lightweight mitigation methods.

[0087] (1) Evaluation experiment of the "real-time detection" step: To evaluate the effectiveness of real-time detection mechanisms from a developer's perspective, a user study was conducted. This study focused on two dimensions: ① the practicality of providing version-specific API signatures during development; and ② the importance of real-time detection (ITAU). Eight graduate students with at least two years of Python development experience were recruited, all of whom were participating in at least one collaborative development project that extensively used third-party libraries. Each participant completed three development tasks selected from real-world open-source projects, covering three typical API evolution types (API name change, parameter change, and return value type change) to ensure comprehensive evaluation. Furthermore, all participants completed the same set of tasks to ensure comparability of evaluation results. The entire experiment was conducted in a virtual IDE environment based on VS Code and integrated with the real-time detection module. After completing the development tasks, participants completed a questionnaire using a five-point Likert scale (1 = useless, 5 = very useful) to rate the "practicality of version-specific API signatures" and "importance of real-time detection (ITAU)."

[0088] The evaluation results show that the average score for API signature usability was 4.63, and the average score for the importance of real-time incompatibility detection was 4.75. Participant feedback indicated that the immediate presentation of standardized API signatures not only reduced the time spent consulting external documentation but also significantly increased developers' confidence in using the API correctly. Furthermore, the evaluation emphasized that "real-time detection" can effectively prevent hidden runtime errors that are difficult to track in large projects. Experiments demonstrated that the "real-time detection" mechanism can significantly improve development efficiency and code reliability with almost no increase in runtime overhead.

[0089] (2) Evaluation experiment of "lightweight relief" steps: To evaluate the effectiveness of the lightweight repair mechanism, the prompt words in RQ1 that cause the model to generate ITAU were selected as the test level, and the focus was on API-level completion where incompatibility could be clearly traced back to specific third-party library API calls. The "Repair Success Rate" (RSR) was defined as the percentage of tasks that, after lightweight repair, were successfully converted into compatible calls according to the compatibility rules defined in Table 3.

[0090] Table 7 summarizes the lightweight fix evaluation results: the RSR of all tested models ranged from 2.96% to 46.34%. DeepSeek-Coder performed best, with an RSR as high as 46.34%, demonstrating strong adaptability to the prompt signature injection strategy. It should be noted that only the samples that actually generated ITAU in RQ1 for each model were fixed; therefore, the fixed sample sets for different models are not the same, and the absolute value of the RSR should not be directly used for performance comparison between models. Experimental results show that the lightweight fix mechanism proposed in this invention can effectively guide large language models to generate API calls consistent with the target version, effectively alleviating compatibility issues.

[0091] Table 7: RSR of each model under the lightweight repair mechanism In summary, this invention focuses on the ITAU problem in LLM-driven code completion results, and analyzes the core challenges brought about by dynamic dependency constraints and fine-grained API evolution. Evaluation results show that, under the above conditions, all tested models struggle to maintain stable consistency between API calls and the actual library version. To address this issue, this invention proposes two lightweight intervention strategies: "real-time detection" and "lightweight repair." The former accurately identifies incompatible calls, while the latter only performs local regeneration on affected call points, avoiding global code refactoring. These findings provide a practical technical path to enhance the compatibility of LLM-driven development environments and point to a promising research direction for integrating "library evolution awareness" capabilities into future code generation systems.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating and mitigating API incompatibility issues in large model code completion, characterized in that, Includes the following steps: S1. Build a knowledge base for third-party library APIs for Python language third-party libraries; S2. To address potential incompatibility issues in third-party library API usage in the code completion code generated by the code completion system based on the large language model, an automatic evaluation is performed based on the third-party library API knowledge base. S3. For the incompatibility issues identified in the assessment, a lightweight intervention mechanism is used to detect and repair them in real time.

2. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 1, characterized in that, Step S1 specifically includes the following sub-steps: S1.1 Define a unified API model to characterize the core structural information of third-party library APIs; S1.2 Select third-party libraries from PyPI and download their source code for each version to your local machine; create a global timeline based on the release timestamps of the third-party libraries, and standardize the version tags of the third-party libraries to obtain PyPI packages; S1.

3. Based on the defined API model, extract API signatures from the source code of each version of the third-party library in the PyPI package. In the process of API signature extraction, a static import flow analysis mechanism is introduced to track the re-import, export and propagation of symbols between Python modules. That is, a directed graph is constructed with modules as nodes and import relationships as edges, and the reachable alias path set of each API is calculated by passing closures. S1.4 Perform cross-version aggregation and difference analysis on each third-party library to detect structural changes, build a version index based on API signatures, and record changes that occur between adjacent versions, including add operations, remove operations, and modify operations, in order to build a third-party library API knowledge base.

3. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 2, characterized in that, The API model is a four-tuple model, which includes name, formal parameters, return type, and scope.

4. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 2, characterized in that, The third-party API knowledge base includes a versioned API signature library for recording all API signatures in each version and an inter-version API difference library for capturing changes between adjacent versions.

5. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 1, characterized in that, Step S2 specifically includes the following sub-steps: S2.1 Collect Python projects that use third-party libraries from GitHub as research corpus, and filter them based on project information and dependency coverage as the final raw data; S2.2 Extract potential third-party library API incompatibility issues from the completion code in Python projects through three stages: third-party library API usage detection, dependency declaration verification and API stability check, and API usage isolation. S2.3 For potential third-party library API incompatibility issues in the completed code, construct the code completion task in the local context of the API call in the file where the third-party library is located. Specifically, the code completion task is constructed through two levels: API-level completion and function-level completion. S2.4 Construct prompt words based on the code completion task, and use a large language model to generate completed Python code based on the prompt words; S2.

5. For the completed Python code, extract the API signatures of its calls, then compare them with the expected API signatures of the target version of the third-party library, and use two API signature-level compatibility evaluation metrics, namely compatibility rate and destructive completion rate, to evaluate the compatibility perception capability of the large language model in the real third-party library evolution scenario.

6. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 5, characterized in that, The third-party library API usage detection specifically includes: performing static analysis on the completion code in the Python project to identify all API calls in the completion code, and standardizing each API call into an FQN containing the full path, module, class name, and API name; identifying the third-party library API calls by matching the FQN with the API signatures in the third-party library API knowledge base; The dependency declaration verification and API stability check specifically include: first, obtaining the version range corresponding to each third-party library from the dependency declaration of the Python project; then, extracting the version information of each API call in the third-party library API call and comparing it with all API signatures of the corresponding version range in the third-party library API knowledge base to verify whether the third-party library API call is compatible with the version range specified in the dependency declaration, and identifying third-party library API calls that may cause incompatibility issues in the use of third-party library APIs during the library version evolution process; The API usage isolation specifically includes: constructing a call graph for third-party library API calls obtained after dependency declaration verification and API stability checks; capturing the call chain within the file by traversing the call graph of the current file to isolate the third-party library API calls within the scope of the current file, thereby obtaining the final potential third-party library API usage incompatibility issues.

7. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 5, characterized in that, The formula for calculating the compatibility rate is: In the formula, CR represents the compatibility rate. This represents the set of prompts being evaluated. Large Language Model (LLM) targets prompt words The generated completed Python code, This represents the set of completed Python code that is compatible with the API signature of the target version of third-party libraries. This is an indicator function that returns 1 if the condition is true, and 0 otherwise. The formula for calculating the destructive completion rate is as follows: In the formula, Indicates the destructive completion rate. This represents the set of completion results where the completed Python code violates the corresponding API signature.

8. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 1, characterized in that, In step S3, the lightweight intervention mechanism specifically includes the following process: Real-time detection: Continuously monitor the third-party library API calls generated during the code completion process. When the code completion system based on the large language model recognizes a third-party library API call, it immediately queries the third-party library API knowledge base to obtain the API signature for the current version. The system then compares the identified third-party library API calls with the queried API signatures, and marks the API call as a potential incompatibility issue if any destructive rule is violated. At the same time, regardless of whether a potential incompatibility issue is detected, the code completion system based on the large language model will present the queried API signatures to the developers in real time as a reference to help them judge the incompatibility issues of the third-party library API calls in an instant. Lightweight Repair: For real-time detection results, a lightweight repair mechanism is used to fix potential incompatibility issues in the markings.

9. The method for evaluating and mitigating the API incompatibility problem of third-party libraries in large model code completion according to claim 8, characterized in that, The lightweight repair mechanism specifically includes: For each potential incompatibility issue, namely incompatible third-party library API calls, two types of key information are injected into the original prompt words to obtain new prompt words. The new prompt words are then used to guide the large language model to generate compatible third-party library API calls, thereby completing the repair of potential incompatibility issues. The two key types of information include: the type and cause of potential incompatibility issues in the current version of the third-party library, and the API signature of a specific version extracted from the API knowledge base of the third-party library.

Images