Artificial intelligence-based programming agent testing method and device, and related equipment

Abstract

This disclosure provides a method, apparatus, and related equipment for testing programmable intelligent agents based on artificial intelligence, relating to the field of artificial intelligence technology, particularly to the fields of intelligent interaction, deep learning, and large-scale models. The specific implementation scheme is as follows: Extracting the natural language input of the target object and its context information from the interaction records between the target object and the first programmable intelligent agent; determining the code information of the natural language input based on the natural language input and the context information, the code information including the target repository and the code submission version; processing the natural language input and the code information based on a first large-scale model to obtain a binary classification result of the natural language input; and testing a second programmable intelligent agent based on the natural language input when the binary classification result indicates that the natural language input requires code understanding capabilities to respond.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of intelligent interaction, deep learning, and large models. Background Technology

[0002] A coding agent is a processing system for programming capabilities built on a large model. Through the semantic understanding and code generation capabilities provided by the large model, it can understand and respond to natural language related to code development, testing, etc.

[0003] In practical applications, the programmed intelligent agent must be tested in order to transform it into a reliable productivity tool. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and related equipment for testing programmable intelligent agents based on artificial intelligence.

[0005] According to one aspect of this disclosure, a method for testing programming intelligent agents based on artificial intelligence is provided, comprising: Extract the natural language input of the target object and the context information of the natural language input from the interaction record between the target object and the first programming agent; Based on the natural language input and the context information, the code information of the natural language input is determined, and the code information includes the target repository and the code commit version; Based on the first major model, the natural language input and the code information are processed to obtain a binary classification result of the natural language input; If the binary classification result indicates that the natural language input needs to be answered through code understanding, a second programming agent is tested based on the natural language input.

[0006] According to another aspect of this disclosure, an artificial intelligence-based programming agent testing apparatus is provided, comprising: The extraction module is used to extract the natural language input of the target object and the context information of the natural language input from the interaction record between the target object and the first programming agent. The determination module is used to determine the code information of the natural language input based on the natural language input and the context information, wherein the code information includes the target repository and the code submission version; The classification module is used to process the natural language input and the code information based on the first large model to obtain a binary classification result of the natural language input; The testing module is used to test a second programming agent based on the natural language input when the binary classification result indicates that the natural language input needs to be answered through code understanding.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a flowchart illustrating a method for testing an AI-based programming intelligent agent according to an embodiment of the present disclosure. Figure 2 This is another flowchart illustrating a method for testing an AI-based programming agent according to an embodiment of the present disclosure; Figure 3 This is another flowchart illustrating a method for testing an AI-based programming agent according to an embodiment of the present disclosure; Figure 4 This is another flowchart illustrating a method for testing an AI-based programming agent according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of a test method for an artificial intelligence-based programming intelligent agent according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the architecture of a programming intelligent agent testing method based on artificial intelligence according to an embodiment of the present disclosure; Figure 7This is a schematic diagram of the structure of an artificial intelligence-based programming agent testing device according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram of the structure of an artificial intelligence-based programming agent testing device according to an embodiment of the present disclosure; Figure 9 This is a block diagram of an electronic device used to implement the AI-based programming agent testing method of the present disclosure embodiments. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0014] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0015] In related technologies, the testing of programming agents relies solely on expert experience, with experts constructing the test questions. This results in low testing efficiency and fails to cover actual testing needs. Furthermore, the existing solutions do not implement a screening mechanism to determine whether access to the code is necessary for a correct answer. This leads to a large number of questions that can be answered using parameterized knowledge, causing the evaluation results to fail to effectively reflect the programming agent's true understanding of the code logic.

[0016] In view of this, embodiments of this disclosure provide a testing method for a programmable intelligent agent based on artificial intelligence. For example... Figure 1 The flowchart shown below illustrates the method and includes the following: S101, extract the natural language input of the target object and the context information of the natural language input from the interaction record between the target object and the first programming agent.

[0017] Among them, programming intelligent agents are systems built on large language models that can understand natural language instructions and execute code-related tasks.

[0018] Natural language input refers to code-related questions or instructions posed by the target object (usually a user) in natural language. For example, text input or voice input from the target object can be processed as natural language input.

[0019] The contextual information of the natural language input consists of explicitly specified code queried by the target object through at least one round of natural language input, or additional data associated with that code, including but not limited to: the file path opened by the user in the IDE (IDE_opened_file), the code snippet selected by the user in the IDE (IDE_selection), the list of code blocks referenced by the user in the message (code_snippets, which may contain multiple), the list of related file paths (file_paths), the session identifier (session_id), the timestamp (create_time), and the source channel (channel). The timestamp indicates the creation time of the current round of natural language input, and the source channel expresses the source of the natural language input; in practice, it can be expressed using the identifier of the first programming agent. This information supplements the code context not explicitly stated in the natural language input.

[0020] S102, based on natural language input and context information, determines the code information of the natural language input, including the target repository and code commit version.

[0021] The target repository is the code repository (Repo) involved in the natural language input of the target object. A code repository is a storage entity in a version control system used to centrally store, manage, and track source code and its change history. It contains a complete collection of project files, a history of all commit versions, branch and tag information, metadata (such as committer, timestamp, and change notes), and version topology (such as the directed acyclic graph structure in Git).

[0022] A commit version is a unique identifier created by a developer after persisting their local code changes to a repository within a version control system (such as Git or SVN (Subversion)). For example, it can be represented by a 40-bit SHA-1 hash value of a Git commit, used to uniquely determine the code's state.

[0023] The code information is in the form of structured triples: repo (the name of the target repository), base_commit (the version of the code commit), and match_source (the matching strategy flag).

[0024] S103, based on the first large model, processes natural language input and code information to obtain binary classification results of natural language input.

[0025] The first major type of model can be Large Language Models (LLMs). Large language models refer to specific types of large-scale models specifically designed for processing text data. These models are neural network-based natural language processing models that can be used to generate, understand, and process text data. Large language models can have hundreds of billions of parameters, generate high-quality text, and can be used for various natural language processing tasks, such as question answering, text generation, and dialogue systems.

[0026] Large language models possess excellent reasoning and few-shot learning capabilities. This is because large language models are comprehensible in that they can be trained on a large number of samples. Based on such large models, accurate semantic understanding can be achieved.

[0027] In addition, the first major model can also be a multimodal large language model (MLLM). When the natural language input of the target object is in image format, it can be processed based on a multimodal large language model.

[0028] The binary classification result of the natural language input is used to indicate whether answering the natural language input requires actual understanding of the code information. That is, answering the natural language input of the target object cannot rely solely on the general knowledge and evaluation experience learned by the large model, but must rely on the ability to understand and reason about the code specified by the natural language input.

[0029] S104, In cases where the binary classification result indicates that the code understanding ability is required to answer the natural language input, test the second programming agent based on the natural language input.

[0030] The second programming agent is the one that needs to be tested. It can be the first programming agent, or a different model version of the first programming agent, a different vendor implementation, or a variant obtained through different training strategies. It can also be any programming agent.

[0031] In the case where there are multiple rounds of dialogue in the interaction record between the target object and the first programming agent, the input of the target object in each round can be processed as an independent candidate sample. Alternatively, the inputs of multiple rounds in the same session can be merged into a task unit according to actual needs (for example, the target object corrects or supplements the question of the previous round in subsequent rounds). However, regardless of the method used, each candidate sample independently executes the complete process of S101-S104.

[0032] In this embodiment, natural language input and its context are extracted from real interaction records, and binary classification is performed using a first-class model. This aligns the type distribution of test questions with real user needs, reducing subjective bias caused by manually constructed questions and improving the representativeness and predictive effectiveness of test results for real-world application scenarios. Furthermore, code information is determined for each natural language input before testing, and binary classification is used to identify whether the input requires code comprehension capabilities from the programming agent. This allows for the discovery of test samples that effectively distinguish between code reasoning ability and parameterized knowledge memorization, enabling the evaluation of true code logic comprehension and ultimately improving the accuracy of the evaluation results. Moreover, the solution proposed in this embodiment requires no additional manual question writing and can be automated, exhibiting good scalability.

[0033] Extracting the natural language input of the target object and the contextual information of the natural language input from the interaction records between the target object and the first programmed intelligent agent can be implemented as follows: Step A1: Extract the dialogue history of each round of dialogue from the interaction records between the target object and the first programmed intelligent agent.

[0034] In implementation, for each round of dialogue, the dialogue history can include the natural language input of the target object in that round of dialogue, and the response information of the first programming agent in that round of dialogue. In addition, information on the first programming agent's thought process in that round of dialogue can be included as needed.

[0035] Step A2: Extract the user's original input and the original context of the user's original input from the dialogue history.

[0036] Raw user input represents the original natural language text entered by the target object in the dialogue, without any preprocessing or rewriting.

[0037] The original context refers to the original context information obtained for this round of dialogue based on the user's original input. This original context information may include key code-related information associated with the user's original input, such as the code repository, code snippets, and related files required to complete this round of dialogue.

[0038] Step A3: If the user's original input contains a technically oriented description and the original context contains code information, generate the natural language input and context information of the target object based on this round of dialogue.

[0039] Technical descriptions refer to specific expressions related to technology contained in natural language input, such as function names, class names, API names, error codes, file path fragments, and programming terms (e.g., compile, error reporting, call, return value, etc.). This technical description indicates that the input to the target object contains code-related technical content.

[0040] Code information represents code-related content contained in the original context. It is used to determine whether the natural language input refers to actual, existing code that needs analysis, and to answer the target user's natural language input based on the analysis results. For example, `ide_opened_file` and `code_snippets` can serve as criteria for determining the presence of code information. In implementation, specific automatic content is extracted from the user's original context. If `ide_opened_file` or `code_snippets` is empty, it indicates that the original context does not contain code information, and the user's original input needs to be deleted. If `ide_opened_file` or `code_snippets` in the user's original input are not empty, then it is determined that the original context contains code information.

[0041] When a user interacts with the first programmed agent, non-technical input may occur (such as "hello," "thank you," "try again," etc.). By detecting technically oriented descriptions, invalid information can be quickly filtered out. If the user's original input contains technically oriented descriptions and the original context contains code information, the natural language input and context information of the target object are generated based on that round of dialogue. If the user's original input does not contain technically oriented descriptions or the original context does not contain code information, the user's original input is filtered.

[0042] In this embodiment of the disclosure, by detecting whether the natural language input of the target object contains technically oriented descriptions, technical dialogues can be distinguished from non-technical dialogues (such as daily greetings, confirmatory replies, etc.), thereby reducing the computational resource consumption caused by invalid natural language in the first major model classification process. Furthermore, by detecting whether the original context contains code information, the basic conditions for responding based on actual code are met by obtaining natural language input and context information, avoiding situations where test samples cannot be constructed due to a lack of code information.

[0043] Before generating the natural language input for the target object based on the user's original input in this round of dialogue, it can also be implemented by: determining that the number of characters contained in the user's original input is greater than a preset threshold.

[0044] If the number of characters in the user's original input exceeds a preset threshold, it is used as the natural language input of the target object, and the operation of generating the natural language input and context information of the target object in step A3 continues. If the number of characters in the user's original input does not exceed the preset threshold, the user's original input is discarded.

[0045] In this embodiment, by setting a character count threshold, excessively short invalid inputs are filtered out, reducing the interference of invalid user inputs on the data processing flow. This also ensures that the retained user inputs have sufficient information density, reducing the possibility of low-confidence classification results due to insufficient input information during subsequent processing by the first major model. In other words, by improving the quality of test samples, the testing accuracy of the second programmed agent is enhanced.

[0046] In practice, some user intentions can be revealed in a single round of dialogue, but many user intentions actually require multiple rounds of dialogue to determine. To improve the quality of test samples, the natural language input and contextual information of the target object are generated based on this round of dialogue. This can be implemented as follows: querying multi-turn dialogues from the interaction record based on the user's original input; wherein the multi-turn dialogues have a referential relationship with the entity nouns in the user's original input; fusing the multi-turn dialogues to obtain the natural language input; and extracting the contextual information of the natural language input from the dialogue history of the multi-turn dialogues.

[0047] Entity nouns include, but are not limited to, pronouns, abbreviations, or abbreviations that omit context. Examples of pronouns include "it," "this," and "that," while abbreviations such as p / d represent prefill / decode.

[0048] This referential relationship refers to the semantic association between an entity noun appearing in the current round and a specific code entity (such as a function name, class name, file path, variable name, API name, etc.) in a previous round.

[0049] The retrieved multi-turn dialogues are fused chronologically to complete the information omitted or ambiguous in the user's original input in the current turn, resulting in semantically complete natural language input. Fusion includes integrating entity-related descriptions from previous turns (such as definitions of code entities, error descriptions, and functional specifications) into the current turn, ensuring that the current natural language input remains fully understandable even when removed from the original dialogue context. Fusion can also be implemented by concatenating multi-turn dialogues. Furthermore, fusion can also be implemented by concatenating multi-turn dialogues and then rewriting them to obtain fluent and clearly defined natural language input.

[0050] From the dialogue history of multi-turn conversations, extract contextual information corresponding to the natural language input. This information collectively constitutes the code environment information of the natural language input, which is used for subsequent processing.

[0051] In this embodiment, by querying multi-turn dialogues with referential relationships based on entity nouns, the historical dialogue content corresponding to referential expressions such as pronouns and abbreviations in the current turn can be located, providing a data foundation for referential resolution. Furthermore, by fusing the queried multi-turn dialogues to complete the information omitted in the user input of the current turn, the complete semantics of the target object, which can only be clearly expressed through multi-turn dialogue, are restored. Simultaneously, by extracting contextual information from the history of multi-turn dialogues, the final generated contextual information can cover multiple code entities and files involved in the multi-turn dialogues, thereby improving the quality of test samples and increasing the testing accuracy of the second programming agent.

[0052] To further improve the quality of automatically selected test samples and reduce the consumption of underlying hardware resources such as computing resources in subsequent processing, after extracting the natural language input of the target object and the context information of the natural language input based on the aforementioned method, the following implementation can be adopted: input the natural language input and context information into the second model to obtain the coarse classification result and coarse classification confidence of the second model for the natural language input; if the coarse classification result indicates that the question intent of the natural language input is to understand code, and the coarse classification confidence is greater than the confidence threshold, input the natural language into the third model so that the third model can perform fine classification of the natural language input with reference to the fine classification set to obtain the sub-capability labels of code understanding ability required to answer the natural language input; the fine classification set includes at least one of the following: logical reasoning ability, architectural understanding ability, defect attribution ability, impact surface analysis ability, and code review ability.

[0053] Among them, the first major model, the second major model, the third major model, and subsequent major models such as the fourth major model, the fifth major model, and the sixth major model can be different roles corresponding to the major models with the same structure, or they can be major models with different structures. This disclosure does not limit them in this regard.

[0054] Specifically, the natural language input and context information mentioned in step S101 are input into the second main model. The second main model performs coarse-grained classification on the natural language input, outputting the coarse classification result and the corresponding coarse classification confidence score. The coarse classification result is used to indicate whether the question intent of the natural language input is a code-understanding category. If it is determined that the question intent of the natural language input does not belong to the code-understanding category or the coarse classification confidence score is not greater than the confidence threshold, it is filtered out and not used as a test sample. If it is determined that the question intent of the natural language input belongs to the code-understanding category and the coarse classification confidence score is greater than the confidence threshold, the natural language input and context information are input into the third main model. The third main model performs fine classification on the natural language input with reference to the fine classification set. The third main model selects one or more sub-capability types suitable for answering the natural language input from the fine classification set as the fine classification result of the natural language input, that is, to obtain the sub-capability label.

[0055] Among them, logical reasoning ability (logic_understanding): This ability mainly assesses the ability to logically reason about code based on natural language input, such as the ability to reason about at least one of the following: function call chains, control flow paths, concurrent scheduling mechanisms, and memory models. For example, the execution order problem of asynchronous scheduling can test logical reasoning ability.

[0056] Architectural understanding ability: This ability assesses the capacity to understand at least one of the following: cross-file, cross-module interface protocols, layered design and data flow, and overall system architecture. Example: "Please describe..." This is the overall design logic for a set of related processes that require collaborative work. Bug analysis: This ability assesses the capacity to locate and analyze the root cause of program errors or abnormal behavior by considering at least one of the following: code implementation, execution environment, and boundary conditions. A typical example is asking: Why does a call to a specific code within a particular loading method in a code repository cause an exception?

[0057] Impact analysis capability: This capability examines the ability to reverse-track dependencies between modules and predict the scope of impact of changes.

[0058] The ability to perform code review (impact analysis) assesses the technical rationale for code changes (diff) within a diff context.

[0059] In this embodiment, the second major model performs coarse classification and confidence filtering to remove natural language inputs with unclear intent or low confidence, as well as contextual information, before proceeding to fine classification. This reduces the computational cost of subsequent fine classification and improves overall processing efficiency. The third major model then performs fine classification based on the set of fine classifications, labeling each natural language input with the specific sub-capability type required. This provides test samples for the finely segmented capabilities when testing the second programming agent, allowing for testing of different sub-capabilities.

[0060] After filtering based on the aforementioned steps, the target object's natural language input and context information are obtained. To improve the quality of the test samples, the code information of the natural language input is determined based on the natural language input and context information. This can be implemented by: matching the natural language input and context information with preset repository features to obtain the target repository; and matching the context information with preset version features to obtain the code submission version.

[0061] The preset repository features are predefined matching criteria used to identify the target repository, including but not limited to: explicit repository name, file path features, code identifiers, and unique terms in the document.

[0062] The preset version features are predefined matching criteria used to identify code submission versions, including but not limited to: user question timestamps, PR (Pull Request) related information, code snippet content, etc.

[0063] Natural language input and its contextual information contain a variety of information that can be used to identify the target repository and code commit version. By matching this information with preset repository and version characteristics, the code information corresponding to each natural language input can be automatically determined without human intervention. This provides accurate code environment anchoring for subsequent automatic selection of test samples and subsequent testing. Decoupling repository matching and version matching into two independent matching processes allows each to adopt the most suitable matching strategy.

[0064] In this embodiment of the disclosure, the target repository is automatically determined by matching natural language input and context information with preset repository features, without the need for manual specification, thereby improving processing efficiency. By matching context information with preset version features, the code commit version can be automatically determined, so that each natural language input is fixed on a unique code snapshot, reducing the problem of non-unique code information caused by continuous repository evolution.

[0065] The natural language input and context information are matched with preset warehouse features to obtain the target warehouse. This can be implemented as follows: query the warehouse name in the natural language input and context information, and determine the queried warehouse name as the target warehouse.

[0066] When a target user interacts with a programming agent, they may sometimes directly mention the name of a code repository in the raw input or text. By explicitly querying the corresponding repository name, the target repository can be quickly identified with minimal computational cost. This approach is most direct and efficient when the user has explicitly mentioned the repository name, eliminating the need for complex feature fusion or candidate repository traversal.

[0067] Specifically, the method for matching natural language input and / or contextual information with preset repository features to determine the target repository is as follows: The repository name is searched for in the natural language input and / or contextual information. The repository name includes, but is not limited to, the full format owner / repo, abbreviations, etc. If the repository name is found in the code block list in the natural language input or contextual information, that repository name is directly identified as the target repository; if no repository name is found, the target repository can be further determined based on other methods, which will be explained later.

[0068] In this embodiment of the disclosure, the warehouse name is queried directly from the natural language input and context information, and fast matching is achieved in scenarios where the user has explicitly mentioned the warehouse name. No additional feature extraction or candidate warehouse traversal is required, thereby reducing the computational overhead of matching.

[0069] The natural language input and context information are matched with preset repository features to obtain the target repository. Alternatively, the process can be implemented as follows: query candidate paths with repository features in the context information, compare the candidate paths with the candidate repository file tree, and obtain the candidate repository that matches the candidate path as the target repository.

[0070] Among them, the candidate repository file tree is a pre-built tree index that reflects the file directory structure in the candidate repository, which is used to quickly compare the matching relationship between path fragments and actual file paths in the repository.

[0071] When a user interacts with a programming agent, the context information typically includes a list of relevant file paths or a list of code blocks referenced by the user in the message. This is often repository-specific; different code repositories, even those containing files with similar names, will have different path structures. By comparing candidate paths with the candidate repository file tree, these differences in path structure can be used to accurately identify the target repository. This method is particularly effective when the user has not explicitly mentioned the repository name but has already opened or referenced the code file.

[0072] Specifically, the context information is used to query candidate paths with repository characteristics. Candidate paths are path fragments with repository-specific characteristics contained in code_snippets or file_paths. The queried candidate paths are matched with the file tree of the candidate repositories by path prefix matching or path fragment matching. If the candidate path can match the actual path in the candidate repository file tree, the candidate repository is determined as the target repository. If the candidate path does not match the actual path in the candidate repository file tree, the target repository is determined based on other methods, which will be explained later.

[0073] In this embodiment of the disclosure, the target repository is identified by utilizing file path information in the user context. In scenarios where the user does not explicitly mention the repository name, the technical path for automatically determining the repository can be provided. By comparing the candidate path with the candidate repository file tree, the path structure differences between different code repositories are used to achieve accurate matching, thereby improving the accuracy of repository identification.

[0074] The natural language input and context information are matched with preset repository features to obtain the target repository. This can also be implemented as follows: extracting code identifiers from the context information; searching for code identifiers in a candidate repository set; semantically matching the candidate repositories that match the code identifiers with the code in the context information; if the semantic matching degree is higher than a matching degree threshold, the candidate repository is determined as the target repository. The matching degree threshold can be determined based on actual conditions, and this disclosure does not limit it.

[0075] Code identifiers are names used to uniquely identify specific program elements in the code, including but not limited to function names, class names, constant names, etc.

[0076] Code snippets in user context information typically contain one or more code identifiers (function names, class names, etc.). When the same code identifier exists in multiple candidate repositories, the existence of the code identifier alone is insufficient to uniquely determine the target repository. For example, the `forward` function is defined in many deep learning frameworks. In this case, it is necessary to further verify whether the semantic context of the code identifier matches, that is, whether the code structure surrounding the code identifier in the user code snippet is consistent with that in the candidate repositories.

[0077] Semantic matching includes, but is not limited to: comparing the calling method, parameter type, return value characteristics, and module path of code identifiers. Alternatively, it involves reading and understanding code snippets in the context information to obtain a first semantic understanding result; reading and understanding code snippets in the candidate repository that correspond to the code identifiers in the context information to obtain a second semantic understanding result; and comparing the semantic similarity between the first and second semantic understanding results.

[0078] In this embodiment of the disclosure, code identifiers are extracted from context information and searched in candidate repositories. The inherent association between code identifiers and repositories is used to achieve the initial location of the target repository. Based on the semantic matching of code identifiers in candidate repositories, the usage of code identifiers in different code contexts is verified to improve the accuracy of repository matching.

[0079] The natural language input and context information are matched with preset warehouse features to obtain the target warehouse. This can be implemented as follows: determine the document information associated with the natural language input in the context information; obtain candidate warehouses that match the warehouse features in the document information, which are then used as the target warehouses.

[0080] Code repository documentation often contains a wealth of information reflecting the unique technical background of that repository, such as code comments and project documentation. This project documentation includes project-specific terminology, internal module names, and unique abbreviations. When user context information (i.e., code_snippets) contains code comments or project documentation-related content, the unique terminology can be extracted, and a full-text search can be performed on the documents of candidate repositories. The strong correlation between specific terminology and repository documentation can then be used to accurately identify the target repository.

[0081] Specifically, the document information associated with the natural language input is identified from the context information. This document information includes, but is not limited to, code comments, README (readme) text, and design document content. Repository features are then extracted from the document information. These features include, but are not limited to, project-specific terminology, technical names, and project-specific abbreviations. Full-text search matching is performed between these repository features and the documents in the candidate repositories. Candidate repositories that match the repository features in the document information are then identified as the target repositories.

[0082] In this embodiment of the disclosure, by extracting repository features from document information for matching, and by utilizing the correlation between document information and specific code repositories, the target repository is identified in scenarios where there is no explicit repository name and clear file path, thereby improving the efficiency of identifying the target repository.

[0083] Matching context information with preset version characteristics to obtain the code commit version can be implemented as follows: determine the pull request (PR) associated with the natural language input from the context information; and use the merge base commit of the pull request as the code commit version.

[0084] Specifically, pull requests associated with the natural language input are determined from contextual information; these pull requests include, but are not limited to: pull request numbers explicitly referenced in the natural language input, pull requests to which code snippets belong, or pull requests matched by code change content; the merge base point of the pull requests is used as the code commit version.

[0085] A pull request refers to a developer's request in the code repository to merge code changes into the main branch, which includes information such as change descriptions, code differences, and discussion records.

[0086] A merge base commit represents the common ancestor commit of the source and target branches in a pull request. This merge base commit represents the code state upon which the PR is based, belonging to a stable version that already exists in the repository history and does not include changes introduced by the PR. Using a merge base commit as an anchor point allows you to reproduce the original code state that the PR was targeting when the user asked the question.

[0087] In implementation, the `match_source` field is introduced. When the code commit version is obtained based on a pull request, `match_source=pr_match`, indicating that the code commit version is determined through a pull request association strategy. That is, the associated PR is identified from natural language input or contextual information, and the merge base point of that PR is used as the code commit version. Typical scenarios for this flag include: user issues mentioning PR numbers, or code snippets originating from changes made in a specific PR.

[0088] Matching context information with preset version characteristics to obtain the code submission version can be implemented as follows: obtaining the creation time of the natural language input from the context information; filtering out candidate versions from multiple candidate versions that are within a specified time difference range of the creation time as the code submission version.

[0089] The creation time is the timestamp of the natural language input being recorded in the interaction log.

[0090] The specified time difference range is a preset threshold for the time difference between the creation time and the commit time of the candidate version. It is used to determine the acceptable time deviation range when filtering candidate versions. For example, if the time difference range is set to N hours, then candidate versions within N hours of the creation time are considered acceptable. The specified time difference range can be fixed or dynamically determined based on the activity level of the repository.

[0091] For example, when candidate versions are dense (such as in a highly active repository), a smaller specified time difference range can be used for precise positioning; when candidate versions are sparse (such as in a low-activity repository), a larger specified time difference range can be used to avoid matching failures.

[0092] Multiple candidate versions are versions that are some or all of the commits in the commit history of the target repository.

[0093] Specifically, the difference between the commit time and creation time of each candidate version is calculated. If the absolute value of the difference is less than or equal to a specified time difference range, the candidate version is included in the filtering results. Among the multiple candidate versions included in the filtering results, the one with the commit time closest to the creation time is selected as the code commit version.

[0094] When the code commit version is determined based on the creation time, `match_source = timestamp_match` indicates that the code commit version is determined using a time window strategy. That is, the creation time of the natural language input is obtained from the context information, and candidate versions within a specified time difference range from multiple candidate versions are selected as the code commit version. A typical scenario for this flag is: a user question lacks PR-related information but includes a question timestamp, which is used to anchor the latest state of the repository at the time the user asked the question.

[0095] In this embodiment of the disclosure, the creation time of the natural language input is obtained from the context information, and the timestamp is used as the basis for version matching. This reduces the problem of not being able to determine the code submission version when there is a lack of PR association information or semantic matching information. By introducing a specified time difference range, version matching becomes more flexible and applicable.

[0096] When it's impossible to determine the code commit version through time window anchoring (e.g., missing creation time, no candidate version within the time difference range) or PR-related anchoring, the code snippets in the user's context information can be used to compare with historical versions of the repository. Matching the context information with preset version characteristics to obtain the code commit version can be implemented as follows: determine the semantic similarity between the code in the context information and the code of multiple candidate versions; based on semantic similarity, filter the code commit version from the multiple candidate versions.

[0097] Specifically, the code in the context information is converted into a first feature vector, and the corresponding code in each candidate version is converted into a second feature vector. Semantic similarity is calculated between the first feature vector and each second feature vector. The calculated semantic similarity is compared with a preset semantic similarity threshold. If the semantic similarity meets the threshold, the corresponding candidate version is submitted as the code submission version. The semantic similarity threshold can be greater than the threshold, or the semantic similarity can be sorted from largest to smallest, and the candidate version ranked first is submitted as the code submission version.

[0098] When the code commit version is derived from the code in the context information, `match_source=semantic_match` indicates that the code commit version is determined using a semantic similarity strategy. That is, the code in the context information is semantically similar to the code in multiple candidate versions, and the candidate version with the highest semantic similarity is selected as the code commit version. Typical scenarios for this flag include: PR information is unavailable, timestamps are missing, or there are no matching commits within the time window, but the code snippet in the user's context can semantically match a version in the repository's history.

[0099] In this embodiment of the disclosure, by calculating the semantic similarity between the code in the context information and the candidate version code, and by determining the candidate version with the highest semantic similarity as the code submission version, the version matching result is based on the code content itself rather than time or metadata. This enables accurate positioning even in scenarios where repository submission time is irregular or there are a large number of irrelevant submissions, thereby improving the accuracy of code submission versions.

[0100] The natural language input provided by the target object may be unclear. To improve the accuracy of the subsequent processing by the first major model, it can be implemented as follows: The natural language input is rewritten according to the preset rewriting rules; the natural language input can be rewritten by the first large model based on the preset rewriting rules, or by other large models based on the preset rewriting rules, and this disclosure does not limit this.

[0101] The default rewrite rules include at least one of the following: 1) Supplement the full name of the warehouse If the natural language input involves a code repository but does not explicitly provide the full name of the repository, then based on the code information (target repository) determined in the previous steps, the full name of the repository is added to the natural language input so that the natural language input can still clearly point to the repository even after being separated from the determined code information.

[0102] In this embodiment of the disclosure, by supplementing the full name of the warehouse, the natural language input can be clearly directed to the target warehouse in subsequent processing, avoiding incomplete information caused by the lack of warehouse identification.

[0103] Using incomplete repository names can avoid ambiguity and, during subsequent testing, can better reflect the real test scenario, thereby improving the accuracy of test results.

[0104] 2) Replace pronouns and abbreviations in natural language input with technical terms. The pronouns and abbreviations in the natural language input are restored to their corresponding standardized technical terms. The replacement of pronouns is based on the technical entities (such as function names, class names, API names, etc.) identified in the previous steps, while the replacement of abbreviations is based on general technical specifications or the context identified in the previous steps.

[0105] In this embodiment of the disclosure, by replacing pronouns and abbreviations with standardized technical terms, the technical expressions in natural language input are made clearer and more standardized, thereby improving the semantic understanding accuracy of the subsequent binary classification by the first major model.

[0106] In subsequent tests, technical terminology can also be used to reproduce the real test scenario as much as possible, thereby improving the accuracy of the test results.

[0107] 3) Retain the original interrogative tone After performing the above-mentioned supplementation or replacement operations on the natural language input, its original interrogative sentence form (such as interrogative words, question marks, etc.) is maintained. That is, the rewritten text is still presented in the form of a question, and its expression as a question is not changed due to the addition of content.

[0108] In this embodiment of the disclosure, by preserving the tone of the original interrogative sentence, the rewritten text is semantically consistent with the user's original intention, reducing the impact on the accuracy of subsequent classification caused by changes in the question type (such as question, instruction, statement) due to rewriting.

[0109] 4) Prohibit the addition of technical assumptions not mentioned in the natural language input. When performing the above rewriting operations, technical information or inferences not originally included in the natural language input must not be introduced. That is, the rewriting operation is limited to supplementing existing information (such as completing repository names and replacing pronouns) and formatting specifications. It is not allowed to expand or add technical explanations to the user's question based on the knowledge of the large model itself, so as to avoid introducing content beyond the user's original intention in the rewriting result.

[0110] In this embodiment of the disclosure, by prohibiting the addition of technical assumptions, the rewriting operation does not introduce inferential content beyond the user's original intention, thus maintaining the semantic consistency between the rewriting result and the user's original input and reducing subsequent problems caused by excessive rewriting that leads to the test questions deviating from the original intent.

[0111] After rewriting the natural language input, it can also be implemented as: outputting a rewritten description of the natural language input.

[0112] The rewriting description records the specific details of this rewriting operation, including but not limited to: the original input segment being rewritten, the corresponding rewritten text, the rewriting rule identifier applied, and a description of the rewriting location. The rewriting description is associated with the rewritten natural language input and serves as supplementary information for subsequent quality audits or manual verification, facilitating the selection of high-quality test samples.

[0113] In this embodiment of the disclosure, by outputting rewriting instructions, the specific content of the rewriting operation is made traceable, which facilitates the verification of the rationality and accuracy of the rewriting in subsequent data review. In addition, by recording the rewriting location and the rules applied, a basis is provided for manual review, and the cause of the problem can be quickly located when the rewriting result deviates.

[0114] Based on the first major model, natural language input and code information are processed to obtain binary classification results for the natural language input, such as... Figure 2 As shown, it can be implemented as follows: S201, input the natural language input and code information into the first large model to determine whether at least one of the following first set of conditions is satisfied.

[0115] The first set of conditions includes: 1) Correctly responding to natural language input depends on the specific data processing logic of the target repository for the natural language input; If a correct answer requires the specific implementation logic of functions, classes, or modules in the target repository, and this logic is not fully described in public documentation or mainstream technical materials, then it requires specific data processing logic.

[0116] 2) Correctly responding to natural language input depends on the runtime behavior information of the target code that the natural language input is targeting; Among them, runtime behavior information depends on at least one of the following: a specific control flow path, a concurrent synchronization mechanism, or boundary condition handling in the code, in order to answer correctly.

[0117] 3) Correctly answering natural language input depends on the static organizational structure of the target repository, and the static organizational structure is determined by understanding the source code of the target code.

[0118] The static organizational structure of the target repository includes at least one of the following: the file organization structure representing the correct answer to the target repository, the interface between modules, and the dependency relationship, which must be confirmed by reading the code.

[0119] The binary classification results are labeled with Code Cognition Access Necessity (CAN). This aims to ensure that each natural language input and its associated code information substantially tests the code reasoning ability of the first main model. In other words, a correct answer must be obtained through understanding and reasoning about the target code (i.e., satisfying at least one condition in the first condition set), and cannot be achieved solely through the parameterized memory of the first main model. This screening ensures that the evaluation results accurately reflect the true level of the second programming agent's code understanding ability, rather than being influenced by interference from parameterized knowledge memory.

[0120] S202, perform binary classification on the natural language input based on whether at least one condition is met, and obtain the binary classification result; Specifically, if at least one condition in the first set of conditions is met, the binary classification result is a positive example, which can be understood as requiring code comprehension ability to answer; if none of the conditions in the first set of conditions are met, the binary classification result is a negative example, which can be understood as not requiring code comprehension ability to answer.

[0121] In this embodiment of the disclosure, by using dependence on specific data processing logic as one of the judgment conditions, questions that require understanding of code implementation details such as function call chains, control flow paths, and data transformation processes can be effectively identified, preventing such questions from being misjudged as requiring no code understanding ability to answer. By using dependence on runtime behavior information as one of the judgment conditions, questions that require dynamic information such as program execution results, exception stacks, and performance can be effectively identified, ensuring that such questions that depend on the actual running state of the code are correctly classified as requiring code understanding ability. By using dependence on static organizational structure as one of the judgment conditions, questions that require understanding of code architecture such as file directory levels, inter-module interface protocols, and dependency relationships can be effectively identified, ensuring that such questions that require extracting information from the source code organizational structure are correctly filtered. Through the comprehensive judgment of the first set of conditions, a formal judgment on whether code understanding ability is required is achieved from multiple dimensions, avoiding the erroneous inclusion of questions that can be answered solely by general programming knowledge or model parameterization memorization into the code understanding test set, so that subsequent testing of the second programming agent can accurately focus on task types that truly require code understanding ability.

[0122] The binary classification result can be obtained through a direct determination method. The natural language input is directly fed into the first large model, which then performs inference analysis and directly determines whether at least one condition in the first condition set is met. Specifically, the natural language input and code information are input into the first large model to determine whether the natural language input requires at least one condition from the first condition set. Based on whether at least one condition is met, the natural language input is binary classified to obtain the binary classification result.

[0123] Alternatively, an indirect decision-making method can be used. This involves first having the first large model attempt to respond to the natural language input, obtaining its operational process information, and then using this information to determine whether the conditions in the condition set are met. Specifically, the natural language input and code information are input into the first large model to obtain its operational process information; this information is then input into the first large model to determine whether responding to the natural language input requires at least one condition from the first condition set; based on whether at least one condition is met, the natural language input is binary classified to obtain the binary classification result.

[0124] The first model generates specific operational process information during its attempt to answer questions. Making judgments based on this information is more interpretable and traceable than direct judgments, and it also reduces the possibility of misjudgments due to a lack of specific reasoning paths. When the first model finds an explicit indication during its operation that it needs to retrieve code, read files, or analyze code structure, it can be determined that the question indeed relies on code comprehension.

[0125] To ensure that the evaluation of the second programming agent is conducted in a controllable and reproducible environment, the code of the target repository needs to be deployed to an independent evaluation container under a specific commit version. Therefore, testing the second programming agent based on natural language input can be implemented as follows: mount the code of the target repository and the code commit version to the evaluation container; run the second programming agent in the evaluation container and obtain the evaluation results for the second programming agent.

[0126] The evaluation container is a software container used to isolate and run a second programmed intelligent agent.

[0127] Specifically, for each unique (repo, base_commit) obtained above, execute `git clone + git checkout` (clone the code repository and switch to the specified commit version), mounting the complete code of the repository in the specified commit state to the evaluation container as a read-only Docker Volume. Containers are named in the format of a fixed prefix-problem number-short name of the target repository, for example, `case-{idx:03d}-{repo_short}`, where `case` represents the fixed prefix, `idx:03d` represents the problem number (three-digit number, padded with leading zeros if less than three digits), and `repo_short` represents the short name of the target repository, used to identify which repository the code mounted in the container comes from. Each container runs independently, and containers are completely isolated from each other.

[0128] The second programmable agent under test possesses the following three types of read-only tools within the container: Glob (file path pattern search): Used to locate related files; Grep (code content search): Used to search for specific keywords or code patterns; Read (full-text file reading): Used to read the details of code implementation.

[0129] In this embodiment of the disclosure, by mounting the code of a specific submission version of the target repository to the evaluation container, the operating environment of the second programming agent is isolated from the outside world, reducing the problems caused by the external environment affecting the evaluation process, thereby improving the stability and accuracy of the evaluation.

[0130] After receiving the response information generated by the second programming agent in response to natural language input, the quality of this response information needs to be evaluated to measure the code comprehension ability of the second programming agent. Since the quality of the response information involves multiple dimensions, manual evaluation alone is inefficient and costly. Therefore, testing the second programming agent based on natural language input, such as... Figure 3 As shown, it can be implemented as follows: S301, input natural language into the second programming agent and obtain the response information of the second programming agent in response to the natural language input.

[0131] The response information is the result obtained by the second programming agent after analyzing and reasoning about the natural language input.

[0132] Understandably, the second programming agent operates within an evaluation container, i.e., a sandbox container. It uses three types of read-only tools—Glob, Grep, and Read—to read code and generate response information based on natural language input. This response information constitutes the complete answer. The tool call sequences and intermediate thought processes within the second programming agent can be recorded separately in log form for behavioral analysis.

[0133] S302, input the natural language input and response information into the fourth model, so that the fourth model generates a first evaluation result of the code reasoning ability for the response information based on at least one of the following dimensions: accuracy of problem understanding, accuracy of technical understanding, completeness of reasoning process, and depth of response.

[0134] After receiving natural language input and response information, the fourth model actively calls Read / Grep to verify the technical statements involving specific code implementations in the response information from at least one of the following dimensions.

[0135] Problem understanding accuracy measures whether the second programming agent correctly understands the core needs and key information points of the user's problem. For this dimension, the fourth model needs to analyze the semantic correspondence between the response information and the natural language input to determine whether the second programming agent understands the user's intent.

[0136] Technical understanding accuracy measures whether the second programming agent correctly understands relevant technical concepts, code logic, and system mechanisms in its responses. For this dimension, the fourth model requires analyzing and comparing the technical facts obtained through container kernel verification with the technical statements in the response information to determine whether the second programming agent understands the user's intent.

[0137] Reasoning process completeness measures whether the reasoning chain of the second programmed agent is complete and whether it covers the necessary intermediate steps required to answer the question. For this dimension, the fourth model needs to analyze whether the response structure is complete and whether it includes the necessary steps from question analysis to conclusion derivation.

[0138] Response depth measures the breadth and depth of the second programming agent's response in terms of technical detail, insight, and comprehensiveness. For this dimension, the fourth model needs to analyze whether the response includes detailed explanations, boundary case analyses, and implementation considerations.

[0139] In this embodiment of the disclosure, the response information is evaluated from multiple dimensions using a fourth model, thereby achieving an automated assessment of the code comprehension ability of the second programming agent. This eliminates the need for manual annotation of each line, reducing assessment costs and improving the accuracy of the automated assessment.

[0140] For example, the architecture diagram for evaluating the second programmed intelligent agent is as follows: Figure 5As shown, the process includes: mounting the code of the target repository at a specific code commit version to a version anchoring sandbox in read-only mode. Natural language input is given to the second programming agent, which reads the code within the version anchoring sandbox using three read-only tools: Glob, Grep, and Read, and generates response information based on the natural language input. CC Judge, as the fourth model, receives both the natural language input and the response information, and also has read-only access to the target repository. It independently verifies the technical statements in the response by calling the Glob / Grep / Read tools, obtaining the first evaluation result. CC Judge outputs a score based on four dimensions: accuracy of problem understanding, accuracy of technical understanding, completeness of the reasoning process, and depth of the response.

[0141] The evaluation of the aforementioned fourth model is based on an overall impression and can comprehensively assess the fluency, logic, and depth of the answer. However, it may be inaccurate due to the illusion problem of large models. Therefore, in this embodiment of the disclosure, as... Figure 4 As shown, it can also be implemented as follows: S401, based on the fifth model, processes natural language input, code information, and context information according to the following steps to obtain the evaluation question; S4011, Filter factual knowledge from the target code targeted by the natural language input; Factual knowledge includes, but is not limited to, independently verifiable code facts such as function signatures, constant values, conditional branches, return value types, and the names of called sub-functions.

[0142] S4012, generate corresponding evaluation questions for each piece of factual knowledge; During implementation, a corresponding evaluation question can be generated for each piece of factual knowledge. The evaluation question is used to verify whether the response of the second programming agent covers or correctly responds to the factual knowledge.

[0143] S402, input the evaluation questions and response information into the sixth model to obtain the second evaluation result.

[0144] The second evaluation result is an atomized item-by-item verification result, which is used to indicate whether the response information covers the factual knowledge corresponding to the evaluation question, and whether the coverage is accurate.

[0145] S403, cross-validate the first evaluation result and the second evaluation result to obtain the target evaluation result of the second programmed intelligent agent.

[0146] Cross-validation can be implemented based on the Pearson coefficient. For example, the Pearson coefficient of the first and second evaluation results is calculated. If the Pearson coefficient is greater than the correlation threshold, the first and second evaluation results are determined to be highly correlated, and the target evaluation result of the second programmed agent is passed. If the Pearson coefficient is not greater than the correlation threshold, the first and second evaluation results are determined to be not highly correlated, and the target evaluation result of the second programmed agent is failed.

[0147] In this embodiment of the disclosure, by cross-validating the first evaluation result and the second evaluation result, and taking advantage of the complementarity of the two in terms of evaluation granularity, the first evaluation result covers the macro-quality dimension, while the second evaluation result focuses on the accuracy of micro-facts. This enables the target evaluation result to have both comprehensive evaluation capabilities at the macro level and fact verification capabilities at the micro level, thereby improving the reliability of the final evaluation conclusion.

[0148] In summary, the artificial intelligence-based programming agent testing method in this disclosure embodiment, such as... Figure 6 As shown, the process involves five stages: the first stage is the intent acquisition stage; the second stage is the intent classification stage; the third stage is the knowledge anchor binding stage; the fourth stage is semantic normalization and CAN annotation; and the fifth stage is sandbox verifiable evaluation. Specifically, the intent acquisition stage includes extracting the natural language input of the target object and its contextual information; the intent classification stage includes the process of inputting the natural language input and contextual information into the second and third large-scale models for processing; the third stage includes the process of determining the code information of the natural language input based on the natural language input and contextual information; the fourth stage includes the process of rewriting the natural language input and processing the natural language input and code information based on the first large-scale model to obtain a binary classification result of the natural language input; and the fifth stage includes the process of testing the second programming agent based on the natural language input.

[0149] Based on the same technical concept, this disclosure also proposes an artificial intelligence-based programming agent testing device 700, such as... Figure 7 As shown, it includes: The extraction module 701 is used to extract the natural language input of the target object and the context information of the natural language input from the interaction record between the target object and the first programming agent. The determination module 702 is used to determine the code information of the natural language input based on the natural language input and the context information, wherein the code information includes the target repository and the code submission version; The classification module 703 is used to process the natural language input and the code information based on the first large model to obtain a binary classification result of the natural language input; The test module 704 is used to test a second programming agent based on the natural language input when the binary classification result indicates that the natural language input needs to be answered through code understanding ability.

[0150] The AI-based programming intelligent agent testing device 700, wherein the classification module 703, as shown... Figure 8 As shown, it includes: The determining unit 7031 is used to input the natural language input and the code information into the first large model to determine whether at least one of the following first set of conditions is satisfied; Classification unit 7032 is used to perform binary classification on the natural language input based on whether at least one of the conditions is met, and obtain the binary classification result; The first set of conditions includes: 1) Correctly answering the natural language input depends on the specific data processing logic of the target warehouse to which the natural language input is directed; 2) Correctly answering the natural language input depends on the runtime behavior information of the target code to which the natural language input is directed; 3) Correctly answering the natural language input depends on the static organizational structure of the target repository, and the static organizational structure is determined by understanding the source code of the target code.

[0151] The AI-based programming intelligent agent testing device 700, wherein the extraction module 701, as shown... Figure 8 As shown, it includes: The first extraction unit 7011 is used to extract the dialogue history of each round of dialogue from the interaction records between the target object and the first programming intelligent agent; The second extraction unit 7012 is used to extract the user's original input and the original context of the user's original input from the dialogue history. The generation unit 7013 is configured to generate natural language input of the target object and the context information based on the round-robin dialogue, provided that the user's original input contains a technically oriented description and the original context contains code information.

[0152] The AI-based programming intelligent agent testing device 700, such as Figure 8 As shown, it also includes: The character determination module 801 is used to determine, before the generation unit generates the natural language input of the target object based on the user's original input in the round-robin dialogue, that the number of characters contained in the user's original input is greater than a preset threshold.

[0153] The AI-based programming intelligent agent testing device 700, wherein the generation unit 7013 is specifically used for: Based on the user's original input, multi-turn dialogues are queried from the interaction record; wherein, the multi-turn dialogues have a referential relationship with the entity nouns in the user's original input; The natural language input is obtained by fusing the multi-turn dialogues. The contextual information of the natural language input is extracted from the dialogue history of the multi-turn dialogue.

[0154] The AI-based programming intelligent agent testing device 700, such as Figure 8 As shown, it also includes: The coarse classification module 802 is used to input the natural language input and the context information into the second large model to obtain the coarse classification result and coarse classification confidence of the second large model on the natural language input; The fine classification module 803 is used to input the natural language into a third model when the coarse classification result indicates that the question intent of the natural language input is to understand code, and the confidence of the coarse classification is greater than the confidence threshold. This allows the third model to perform fine classification on the natural language input with reference to the fine classification set, thereby obtaining sub-capability labels for the code understanding ability required to answer the natural language input. The fine-classification set includes at least one of the following: Logical reasoning ability, architectural understanding ability, defect attribution ability, impact analysis ability, and code review ability.

[0155] The AI-based programming intelligent agent testing device 700, such as Figure 8 As shown, it also includes: The rewriting module 804 is used to rewrite the natural language input according to preset rewriting rules; The preset rewriting rules include at least one of the following: 1) Add the full name of the warehouse; 2) Replace pronouns and abbreviations in the natural language input with technical terms; 3) Retain the original interrogative tone; 4) Adding technical assumptions not mentioned in the natural language input is prohibited.

[0156] The AI-based programming intelligent agent testing device 700, such asFigure 8 As shown, it also includes: Output module 805 is used to output a rewritten description of the natural language input.

[0157] The AI-based programming intelligent agent testing device 700, wherein, as Figure 8 As shown, the determining module 702 includes: The first matching unit 7021 is used to match the natural language input with the context information and with preset warehouse features to obtain the target warehouse; and, The second matching unit 7022 is used to match the context information with preset version features to obtain the code submission version.

[0158] The AI-based programming agent testing device 700, wherein the first matching unit 7021 is specifically used for: The warehouse name is queried from the natural language input and the context information, and the queried warehouse name is determined as the target warehouse.

[0159] The AI-based programming agent testing device 700, wherein the first matching unit 7021 is specifically used for: The candidate paths with repository characteristics are queried in the context information, and the candidate paths are compared with the candidate repository file tree to obtain the candidate repository that matches the candidate path as the target repository.

[0160] The AI-based programming agent testing device 700, wherein the first matching unit 7021 is specifically used for: Extract the code identifier from the context information; Search for the code identifier in the candidate repository set; Semantically match the candidate repositories in the candidate repository set that match the code identifier with the code in the context information. If the semantic matching degree is higher than the matching degree threshold, the candidate warehouse is determined as the target warehouse.

[0161] The AI-based programming agent testing device 700, wherein the first matching unit 7021 is specifically used for: The document information associated with the natural language input is determined from the context information; Candidate warehouses that match the warehouse features in the document information are obtained as the target warehouse.

[0162] The AI-based programming agent testing device 700, wherein the second matching unit 7022 is specifically used for: The creation time of the natural language input is obtained from the context information; From multiple candidate versions, select the candidate versions whose creation time is within a specified time difference range and use them as the code submission versions.

[0163] The AI-based programming agent testing device 700, wherein the second matching unit 7022 is specifically used for: Determine the semantic similarity between the code in the context information and the code of multiple candidate versions; Based on the semantic similarity, the code submission version is selected from the multiple candidate versions.

[0164] The AI-based programming intelligent agent testing device 700, wherein the testing module 704, as shown... Figure 8 As shown, it includes: The response information acquisition unit 7041 is used to input the natural language into the second programming agent and obtain the response information of the second programming agent in response to the natural language input. Evaluation unit 7042 is used to input the natural language input and the response information into the fourth model, so that the fourth model generates a first evaluation result of the code reasoning ability of the response information based on at least one of the following dimensions: Accuracy of problem understanding, accuracy of technical understanding, completeness of reasoning process, and depth of answer.

[0165] The AI-based programming intelligent agent testing device 700, such as Figure 8 As shown, it also includes: Processing module 806 is used to process the natural language input, the code information, and the context information according to the fifth model in the following steps to obtain the evaluation question; Factual knowledge is filtered out from the target code targeted by the natural language input; For each piece of factual knowledge, a corresponding evaluation question is generated; The auxiliary module 807 is used to input the evaluation question and the response information into the sixth model to obtain the second evaluation result; The verification module 808 is used to cross-verify the first evaluation result and the second evaluation result to obtain the target evaluation result of the second programming agent.

[0166] The AI-based programming intelligent agent testing device 700, wherein, as Figure 9 As shown, the test module 704 includes: Mounting unit 7043 is used to mount the target repository and the code of the code submission version to the evaluation container; The running unit 7044 is used to run the second programmed agent within the evaluation container and obtain evaluation results for the second programmed agent.

[0167] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0168] The acquisition, storage, and application of any type of information, such as user personal information, involved in the technical solutions disclosed herein comply with relevant laws and regulations and do not violate public order and good morals.

[0169] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0170] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like ​ As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0172] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as an AI-based programmable agent testing method. For example, in some embodiments, the AI-based programmable agent testing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the AI-based programmable agent testing method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured, by any other suitable means (e.g., by means of firmware), to perform an AI-based programming agent testing method.

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0180] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0181] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for testing programming intelligent agents based on artificial intelligence, comprising: Extract the natural language input of the target object and the context information of the natural language input from the interaction record between the target object and the first programming agent; Based on the natural language input and the context information, the code information of the natural language input is determined, and the code information includes the target repository and the code commit version; Based on the first major model, the natural language input and the code information are processed to obtain a binary classification result of the natural language input; If the binary classification result indicates that the natural language input needs to be answered through code understanding, a second programming agent is tested based on the natural language input.

2. The method according to claim 1, wherein, The process of processing the natural language input and the code information based on the first large model to obtain the binary classification result of the natural language input includes: The natural language input and the code information are input into the first large model to determine whether at least one of the following first set of conditions is satisfied; Based on whether at least one of the conditions is met, the natural language input is binary classified to obtain the binary classification result; The first set of conditions includes: 1) Correctly answering the natural language input depends on the specific data processing logic of the target warehouse to which the natural language input is directed; 2) Correctly answering the natural language input depends on the runtime behavior information of the target code to which the natural language input is directed; 3) Correctly answering the natural language input depends on the static organizational structure of the target repository, and the static organizational structure is determined by understanding the source code of the target code.

3. The method according to claim 1, wherein, The step of extracting the natural language input of the target object and the context information of the natural language input from the interaction record between the target object and the first programming agent includes: Extract the dialogue history of each round of dialogue from the interaction records between the target object and the first programmed intelligent agent; Extract the user's original input from the dialogue history, as well as the original context of the user's original input; If the user's original input contains a technically oriented description and the original context contains code information, the natural language input of the target object and the context information are generated based on the round-robin dialogue.

4. The method according to claim 3, before generating the natural language input of the target object based on the user's original input in the round-robin dialogue, further includes: It is determined that the number of characters contained in the user's original input is greater than a preset threshold.

5. The method according to claim 3, wherein, The generation of the natural language input of the target object and the context information based on the round-robin dialogue includes: Based on the user's original input, multi-turn dialogues are queried from the interaction record; wherein, the multi-turn dialogues have a referential relationship with the entity nouns in the user's original input; The natural language input is obtained by fusing the multi-turn dialogues. The contextual information of the natural language input is extracted from the dialogue history of the multi-turn dialogue.

6. The method according to any one of claims 1-5, further comprising: The natural language input and the context information are input into the second large model to obtain the coarse classification result and coarse classification confidence of the second large model on the natural language input; If the coarse classification result indicates that the question intent of the natural language input is to understand code, and the confidence of the coarse classification is greater than the confidence threshold, the natural language input is input into the third model, so that the third model refers to the fine classification set to perform fine classification on the natural language input, and obtains sub-capability labels of code understanding ability required to answer the natural language input; The fine-classification set includes at least one of the following: Logical reasoning ability, architectural understanding ability, defect attribution ability, impact analysis ability, and code review ability.

7. The method according to claim 1 or 5, further comprising: The natural language input is rewritten according to preset rewriting rules; The preset rewriting rules include at least one of the following: 1) Add the full name of the warehouse; 2) Replace pronouns and abbreviations in the natural language input with technical terms; 3) Retain the original interrogative tone; 4) Adding technical assumptions not mentioned in the natural language input is prohibited.

8. The method according to claim 7, further comprising: Output a rewritten description of the natural language input.

9. The method according to any one of claims 1-8, wherein, The step of determining the code information of the natural language input based on the natural language input and the context information includes: The natural language input and the context information are matched with preset warehouse features to obtain the target warehouse; and, The context information is matched with preset version characteristics to obtain the code submission version.

10. The method according to claim 9, wherein, The step of matching the natural language input with the context information and preset warehouse features to obtain the target warehouse includes: The warehouse name is queried from the natural language input and the context information, and the queried warehouse name is determined as the target warehouse.

11. The method according to claim 9, wherein, The step of matching the natural language input with the context information and preset warehouse features to obtain the target warehouse includes: The candidate paths with repository characteristics are queried in the context information, and the candidate paths are compared with the candidate repository file tree to obtain the candidate repository that matches the candidate path as the target repository.

12. The method according to claim 9, wherein, The step of matching the natural language input with the context information and preset warehouse features to obtain the target warehouse includes: Extract the code identifier from the context information; Search for the code identifier in the candidate repository set; Semantically match the candidate repositories in the candidate repository set that match the code identifier with the code in the context information. If the semantic matching degree is higher than the matching degree threshold, the candidate warehouse is determined as the target warehouse.

13. The method according to claim 9, wherein, The step of matching the natural language input with the context information and preset warehouse features to obtain the target warehouse includes: The document information associated with the natural language input is determined from the context information; Candidate warehouses that match the warehouse features in the document information are obtained as the target warehouse.

14. The method according to claim 9, wherein, The step of matching the context information with preset version characteristics to obtain the code submission version includes: The creation time of the natural language input is obtained from the context information; From multiple candidate versions, select the candidate versions whose creation time is within a specified time difference range and use them as the code submission versions.

15. The method according to claim 9, wherein, The step of matching the context information with preset version characteristics to obtain the code submission version includes: Determine the semantic similarity between the code in the context information and the code of multiple candidate versions; Based on the semantic similarity, the code submission version is selected from the multiple candidate versions.

16. The method according to claim 1, wherein, The second programming agent tested based on the natural language input includes: The natural language is input into the second programming agent to obtain the response information of the second programming agent in response to the natural language input; The natural language input and the response information are input into the fourth model, so that the fourth model generates a first evaluation result of the code reasoning ability of the response information based on at least one of the following dimensions: Accuracy of problem understanding, accuracy of technical understanding, completeness of reasoning process, and depth of answer.

17. The method of claim 16, further comprising: Based on the fifth model, the natural language input, the code information, and the context information are processed according to the following steps to obtain the evaluation question; Factual knowledge is filtered out from the target code targeted by the natural language input; For each piece of factual knowledge, a corresponding evaluation question is generated; The evaluation question and the response information are input into the sixth model to obtain the second evaluation result; The first evaluation result and the second evaluation result are cross-validated to obtain the target evaluation result of the second programming agent.

18. The method according to claim 1, wherein, The second programming agent tested based on the natural language input includes: Mount the target repository and the code submission version into the evaluation container; The second programmed agent is run within the evaluation container to obtain evaluation results for the second programmed agent.

19. A testing device for a programmable intelligent agent based on artificial intelligence, comprising: The extraction module is used to extract the natural language input of the target object and the context information of the natural language input from the interaction record between the target object and the first programming agent. The determination module is used to determine the code information of the natural language input based on the natural language input and the context information, wherein the code information includes the target repository and the code submission version; The classification module is used to process the natural language input and the code information based on the first large model to obtain a binary classification result of the natural language input; The testing module is used to test a second programming agent based on the natural language input when the binary classification result indicates that the natural language input needs to be answered through code understanding.

20. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-19.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-19.

22. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-19.

Images