Defect report complementing and optimizing method and system based on multi-agent cooperation

Abstract

The invention provides a defect report completion and optimization method and system based on multi-agent cooperation, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining an original defect report; the completion and optimization task of the original defect report is decoupled into three sequentially executed sub-tasks of defect analysis, report completion and quality evaluation; analyzing the input original defect report through an analysis agent, identifying missing key information fields in the original defect report, and generating a complementation suggestion based on the retrieved external knowledge; a completion suggestion output by defect analysis is received through a completion agent, and missing content in the original defect report is generated and completed based on the completion suggestion to obtain a completed report; and evaluating the quality of the completed report through the evaluation agent. A defect report processing flow is divided into three specialized stages of analysis, completion and evaluation through task decoupling, and high-quality and automatic completion and optimization of the defect report are finally realized by adopting a multi-agent cooperation and dynamic iterative optimization mechanism.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for defect report completion and optimization based on multi-agent collaboration. Background Art

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Software defect reports are an important information carrier in software maintenance, and their quality directly determines the efficiency of developers in locating and fixing defects. A high-quality defect report typically contains three core elements: (1) observed behavior, i.e., the abnormal or erroneous phenomena exhibited by the software; (2) expected behavior, i.e., the correct behavior that the software should exhibit under normal circumstances; and (3) reproduction steps, i.e., a series of operational instructions that can reliably reproduce the defect. These elements provide developers with the key context needed to diagnose and fix problems.

[0004] Currently, automated defect report completion and optimization have become a research hotspot in the field of software engineering. Existing technical solutions can be mainly categorized as follows: The first category is based on traditional machine learning methods. Although these methods can identify and complete missing fields, the generated content often suffers from problems such as logical repetition, verbose descriptions, and a lack of key details, resulting in insufficient readability and practicality.

[0005] The second category is based on a single large language model (LLM). While the generated content is more concise and logical, it is still limited by the cognitive load of a single model and is prone to "illusions," ignoring key information such as environmental configuration and boundary conditions, resulting in inaccurate and incomplete content.

[0006] The third category is multi-agent collaboration frameworks (such as MetaGPT and AutoGen), which attempt to solve complex tasks through the division of labor and cooperation among multiple AI agents. However, these frameworks have obvious limitations: either they lack mechanisms for verifying and iteratively optimizing the generated results, making it difficult to guarantee the final quality; or they rely on loosely session-driven collaboration, which is prone to problems such as inconsistent contextual understanding, semantic drift, and inefficiency, and it is difficult to perform optimal role division and interaction design for specific tasks (such as defect report completion).

[0007] In summary, existing technologies for automating defect report completion generally suffer from problems such as low completion quality, lack of reliable iterative optimization, and low efficiency in multi-agent collaboration. Therefore, there is an urgent need for a new technical solution that can systematically address these issues and efficiently generate high-quality, complete defect reports. Summary of the Invention

[0008] To address the aforementioned issues, this invention provides a defect report completion and optimization method and system based on multi-agent collaboration. This invention divides the defect report processing flow into three specialized stages—analysis, completion, and evaluation—through task decoupling. It employs a multi-agent collaboration and dynamic iterative optimization mechanism, with three dedicated large language model agents cooperating and cross-validating to ultimately achieve high-quality, automated defect report completion and optimization.

[0009] The first aspect of this invention provides a defect report completion and optimization method based on multi-agent collaboration, comprising: Obtain the original defect report; The task of completing and optimizing the original defect report is decoupled into three sub-tasks executed sequentially: defect analysis, report completion, and quality assessment. The defect analysis subtask analyzes the input raw defect report through the analysis agent, identifies the missing key information fields, and generates completion suggestions based on the retrieved external knowledge. The report completion subtask receives completion suggestions from the defect analysis output through the completion agent, generates and completes the missing content in the original defect report based on the completion suggestions, and obtains the completed report. The quality assessment subtask evaluates the quality of the completed report through an assessment agent. If the quality assessment passes, the completed and optimized report is output. If the quality assessment fails, feedback information is generated based on the assessment results and sent to the defect analysis subtask and / or the report completion subtask, triggering a new round of defect analysis and / or report completion until the assessment passes or the maximum number of iterations is reached.

[0010] Furthermore, the analysis agent, completion agent, and evaluation agent are all implemented using a large language model, and each agent is configured with its own structured prompts, which include the received information, team description, and job description.

[0011] Furthermore, the suggestions for generating completion based on retrieved external knowledge include: Hierarchical retrieval and matching methods are used to retrieve external knowledge. These methods include: Calculate the similarity between the target defect report and multiple information centroids pre-stored in the knowledge base. The information centroids are generated based on the clustering of different types of key information in historical defect reports. The top-K candidate knowledge groups are selected from the knowledge base based on centroid similarity. Calculate the similarity between the target defect report and specific statement sets in the candidate knowledge group to determine the most relevant target knowledge from the candidate knowledge group.

[0012] Furthermore, the information centroid includes the observed behavior centroid, the expected behavior centroid, and the reproducible steps centroid.

[0013] Furthermore, the analysis agent receives feedback on retrieval requests from the evaluation agent and adjusts the knowledge retrieval strategy based on the feedback to regenerate completion suggestions.

[0014] Furthermore, the completion agent receives modification suggestions from the evaluation agent and optimizes the content of the completed report based on these suggestions. The optimizations include formatting and / or readability.

[0015] Furthermore, the assessment of the quality of the completed report by the evaluation agent specifically includes: The completed report was evaluated from three dimensions: format correctness, content completeness and readability, and information richness. If the assessment fails, determine the problem type: If the issue is related to content semantics or formatting, a modification suggestion will be sent to the completion agent. If the problem is due to missing or insufficient information, a retrieval request is sent to the analysis agent.

[0016] A second aspect of the present invention provides a defect report completion and optimization system based on multi-agent collaboration, comprising: The report acquisition module is used to obtain the original defect report; The task decoupling module is used to decouple the task of completing and optimizing the original defect report into three sub-tasks that are executed sequentially: defect analysis, report completion, and quality assessment. The defect analysis module is used to analyze the input raw defect report through the analysis agent, identify the missing key information fields, and generate completion suggestions based on the retrieved external knowledge; The report completion module is used to receive completion suggestions from the defect analysis output through the completion agent, and generate and complete the missing content in the original defect report based on the completion suggestions to obtain the completed report; The quality assessment module is used to assess the quality of the completed report through an assessment agent; The control module is used to control the defect analysis module and / or report completion module to perform iterative operations based on feedback information from the quality assessment module.

[0017] A third aspect of the present invention provides a defect report completion and optimization apparatus based on multi-agent collaboration, the apparatus including a memory and a processor; the memory is used to store a computer program; the processor is used to implement the above-described defect report completion and optimization method based on multi-agent collaboration when executing the computer program.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described defect report completion and optimization method based on multi-agent collaboration.

[0019] Compared with existing technologies, the defect report completion and optimization method and system based on multi-agent collaboration provided by the present invention has the following beneficial effects: (1) In response to the technical problem of high cognitive load and hallucination when a single model processes complex tasks, this invention breaks down the complex defect report processing process into three independent stages: analysis, completion, and evaluation, and assigns each stage to a dedicated agent. This effectively reduces the cognitive load of each agent and clarifies the boundaries of their responsibilities, thereby achieving the technical effect of improving the reliability of task processing and the accuracy of generated content.

[0020] (2) In response to the technical problem that the lack of key information in defect reports leads to poor performance of traditional retrieval methods, this invention first calculates the similarity with the information centroid for coarse screening, and then compares the specific sentence set for fine screening. This achieves the technical effect of efficiently and accurately retrieving relevant knowledge even when information is missing, and provides reliable information support for high-quality completion.

[0021] (3) In response to the technical problem of lack of effective quality control and iterative optimization mechanism in multi-agent collaboration, this invention provides a dynamic feedback and iterative optimization mechanism as well as an evaluation agent and feedback path triggering mechanism, and establishes a closed-loop process of "analysis, completion and evaluation", which can perform multiple rounds of cross-validation and collaborative optimization on substandard content, thereby achieving the technical effect of progressively improving the quality of defect reports.

[0022] (4) In view of the technical problem that the role division and interaction method of the general multi-agent framework are unclear in specific tasks, the present invention provides structured prompt instructions, which defines clear input, role, responsibility and output requirements for each agent, so that it can accurately understand its own task, thereby achieving the technical effect of greatly improving the efficiency of multi-agent collaboration and the overall output quality. Attached Figure Description

[0023] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0024] Figure 1 This is a flowchart of the defect report completion and optimization method based on multi-agent collaboration provided in Embodiment 1 of the present invention; Figure 2 This is an overall framework diagram of the method provided in Embodiment 1 of the present invention; Figure 3This is an example diagram of the analysis agent instruction settings provided in Embodiment 1 of the present invention; Figure 4 This is a flowchart of the hierarchical retrieval and matching method provided in Embodiment 1 of the present invention; Figure 5 This is an example diagram of the completion proxy instruction settings provided in Embodiment 1 of the present invention; Figure 6 This is an example diagram of the evaluation agent instruction settings provided in Embodiment 1 of the present invention; Figure 7 This is a line graph showing the effect of the number of iterations on the BLEU performance of the model, provided in Embodiment 1 of the present invention. Figure 8 This is a line graph showing the impact of the number of iterations on the performance of the SentenceBERT model, provided in Embodiment 1 of the present invention. Figure 9 This is a line graph showing the effect of the number of iterations on the performance of the ROUGE-L model, provided in Embodiment 1 of the present invention. Figure 10 This is a line graph showing the effect of the number of iterations on the METEOR performance of the model, provided in Embodiment 1 of the present invention. Figure 11 The method provided in Embodiment 1 of this invention completes the bar chart reflecting BLEU performance for various types of content when processing Github and Eclipse datasets; Figure 12 This is a bar chart reflecting the performance of SentenceBERT by completing various types of content when processing Github and Eclipse datasets, provided by the method in Embodiment 1 of the present invention. Figure 13 This is a bar chart reflecting ROUGE-L performance by completing various types of content when processing Github and Eclipse datasets using the method provided in Embodiment 1 of the present invention; Figure 14 This is a bar chart reflecting METEOR performance by completing various types of content when processing Github and Eclipse datasets, provided by the method in Embodiment 1 of this invention. Figure 15 The method provided in Embodiment 1 of this invention completes the bar charts reflecting BLEU performance for various types of content when processing Mozilla and Tomcat datasets; Figure 16 This is a bar chart reflecting the performance of SentenceBERT by completing various types of content when processing Mozilla and Tomcat datasets, provided by the method in Embodiment 1 of this invention. Figure 17 This is a bar chart reflecting ROUGE-L performance by supplementing various types of content when processing Mozilla and Tomcat datasets using the method provided in Embodiment 1 of this invention; Figure 18 This is a bar chart reflecting METEOR performance by supplementing various types of content when processing Mozilla and Tomcat datasets using the method provided in Embodiment 1 of this invention; Figure 19 This is a schematic diagram of a defect report completion and optimization system based on multi-agent collaboration provided in Embodiment 2 of the present invention. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0029] Example 1 like Figure 1 This embodiment provides a defect report completion and optimization method based on multi-agent collaboration, including: S1: Obtain the original defect report; S2: Decouple the task of completing and optimizing the original defect report into three sub-tasks executed sequentially: defect analysis, report completion, and quality assessment; S3: The defect analysis subtask analyzes the input raw defect report through the analysis agent, identifies the missing key information fields, and generates completion suggestions based on the retrieved external knowledge; S4: The report completion subtask receives completion suggestions from the defect analysis output through the completion agent, generates and completes the missing content in the original defect report based on the completion suggestions, and obtains the completed report; S5: The quality assessment subtask evaluates the quality of the completed report through the assessment agent. If the quality assessment passes, the completed and optimized report is output. If the quality assessment fails, feedback information is generated based on the assessment results and sent to the defect analysis subtask and / or report completion subtask to trigger a new round of defect analysis and / or report completion until the assessment passes or the maximum number of iterations is reached.

[0030] Traditional single-model methods, such as traditional machine learning or a single large language model, must simultaneously undertake multiple responsibilities, including identifying missing information, generating content, and judging quality, when handling end-to-end completion tasks. This results in high cognitive load and poor performance; for example, machine learning methods may produce semantic breaks, and large language models may exhibit hallucination phenomena. This invention, through task decoupling and a multi-agent division of labor mechanism in steps S2 to S4, allows each agent to focus on only a single subtask, significantly reducing cognitive load and structurally circumventing the inherent defects of end-to-end single models.

[0031] Existing multi-agent frameworks, such as MetaGPT or AutoGen, typically lack iterative verification mechanisms for the generated results, often ending the process after a single generation, which poses a quality risk. The iterative feedback mechanism (step S5) proposed in this invention uses multiple rounds of cross-validation, with the evaluation agent providing targeted feedback to the corresponding agent for optimization. This effectively controls semantic drift and ensures that the final report is logically coherent and consistent with reality.

[0032] Specifically, the analysis agent, completion agent, and evaluation agent are all implemented using a large language model, and each agent is configured with its own structured prompts, which include the received information, team description, and job description.

[0033] Existing multi-agent frameworks often employ session-driven collaboration, which can easily lead to inconsistencies in context understanding and unclear responsibility boundaries. This invention provides structured prompts to each agent, offering clear input, role, and output guidelines. This allows large language models to accurately define their responsibilities; for example, the evaluation agent only needs to focus on quality judgments without participating in content generation, significantly improving collaboration efficiency. The responsibility descriptions further clarify output specifications, such as requiring the content generated by the completion agent to be consistent with the original context logic. The input scope is limited based on received information; for example, the analysis agent generates suggestions only based on relevant external knowledge, avoiding the generation of inaccurate or unconstrained content by the large language model due to ambiguous roles or unrestricted input, thus improving the reliability of generated content. The team descriptions and responsibility descriptions in the structured prompts can also be tailored to the domain characteristics of defect reports, such as clearly defining observed behavior, expected behavior, and reproduction steps as core elements, preventing general-purpose large language models from generating unprofessional content due to insufficient domain knowledge.

[0034] Specifically, the suggestions for generating completion based on retrieved external knowledge include: Hierarchical retrieval and matching methods are used to retrieve external knowledge. These methods include: Calculate the similarity between the target defect report and multiple information centroids pre-stored in the knowledge base. The information centroids are generated based on the clustering of different types of key information in historical defect reports. The top-K candidate knowledge groups are selected from the knowledge base based on centroid similarity. Calculate the similarity between the target defect report and specific statement sets in the candidate knowledge group to determine the most relevant target knowledge from the candidate knowledge group.

[0035] Traditional retrieval methods, such as text similarity matching or embedding matching, heavily rely on the complete information of the target report. When the original report lacks key fields, problems such as low matching quality and irrelevant search results easily arise. The hierarchical retrieval method adopted in this invention first performs a coarse screening using centroids. Even if a certain type of information is missing, matching can still be achieved through centroids of other types of information (the three types of centroids serve as "prototype representations" of corresponding statement types in the semantic space; for example, each centroid of an observation behavior statement represents a "symptom pattern" of a similar defect). Then, a fine screening is performed using a set of statements to lock in precise knowledge, thus achieving efficient retrieval even in scenarios with missing information. The retrieved target knowledge comes from the clustering summary of historical similar defect reports, rather than random external knowledge. This ensures that the completion suggestions generated by the analysis agent align with the domain logic of the defect report, such as the recurrence patterns of software defects and industry standards for expected behaviors, providing high-quality knowledge basis for subsequent completion agents and avoiding the generation of irrelevant or erroneous completion content. Centroid coarse screening can quickly narrow down the search scope and solve the problem of low efficiency in full-scale search, while sentence set fine screening ensures accurate results and solves the problem of insufficient accuracy in coarse screening results. The combination of the two can meet the efficiency requirements of rapid processing of a large number of reports in open source scenarios, while also taking into account the accuracy of knowledge retrieval.

[0036] Specifically, the information centroid includes the observed behavior centroid, the expected behavior centroid, and the reproducible steps centroid.

[0037] Observed behavior, expected behavior, and reproduction steps are the three core elements of a defect report. The absence of any one of these elements significantly prolongs the repair time; for example, missing reproduction steps prevents developers from reproducing the defect. This invention ensures that the retrieval process always revolves around these three core elements by defining the types of information centroids, avoiding the problem of missing core information in supplementary suggestions due to missing retrieval dimensions. Regardless of which type of core information is missing in the original report, the system can match it using centroids of other types of information. For example, when expected behavior is missing, similar reports can be located using the observed behavior centroid and the reproduction step centroid, thus breaking the limitation of traditional retrieval methods that cannot match due to missing information and improving the method's adaptability to different incomplete reports. Clearly defining the types of information centroids also provides a unified basis for the clustering process of historical defect reports, i.e., clustering according to the three core elements, avoiding the problem of meaningless centroids due to chaotic clustering dimensions, and ensuring the consistency and reliability of subsequent retrievals.

[0038] Specifically, the analysis agent receives retrieval request feedback from the evaluation agent and adjusts the knowledge retrieval strategy based on the retrieval request feedback to regenerate completion suggestions.

[0039] Existing multi-agent methods often suffer from information fragmentation between agents; for example, the evaluation agent may identify problems but fail to effectively provide feedback to the analysis agent. This invention introduces a feedback mechanism, enabling the evaluation agent's quality judgments to directly guide the analysis agent's retrieval behavior. This addresses the problem of insufficient quality in the suggestions generated by the analysis agent without the ability to optimize them, thereby improving the accuracy and richness of the completion suggestions. The analysis agent adjusts its retrieval strategy based on specific feedback from the evaluation agent. For example, when feedback indicates a lack of environment configuration information, it can specifically search for similar reports containing environment configuration information, avoiding blind re-searching, reducing unnecessary computational resource consumption, and improving iteration efficiency. When the evaluation agent determines that the completed content is inconsistent with the facts, possibly due to insufficient or irrelevant knowledge retrieved by the analysis agent, the analysis agent can adjust its strategy to supplement more relevant knowledge, providing a more reliable basis for the completion agent and reducing the risk of generating illusory content based on erroneous knowledge from the source.

[0040] Specifically, the completion agent receives modification suggestions from the evaluation agent and optimizes the content of the completed report based on the modification suggestions. The optimizations include formatting and / or readability.

[0041] Traditional machine learning methods often suffer from repetitive logic and verbose descriptions in their completed reports. For example, reproduction steps based on retrieval and concatenation may be described repeatedly; single large language model methods may ignore formatting specifications, such as outputting unstructured text. This invention optimizes formatting and readability to ensure a clear final report structure. The JSON format facilitates quick review by developers, and the concise, non-redundant content enhances the report's usability. A well-formatted report, including fields such as observed behavior, expected behavior, and reproduction steps, allows developers to quickly locate key information. Readability-optimized content, such as concise reproduction steps, reduces developer comprehension time, indirectly improving defect repair efficiency. By clearly defining the optimization direction—formatting and readability—it ensures that different original reports conform to a unified standard after completion, avoiding the additional cost for developers to adapt to different report styles due to inconsistent formats or descriptions. This is particularly suitable for application scenarios with multiple contributors and projects in open-source communities.

[0042] Specifically, the assessment of the quality of the completed report by the assessment agent includes: The completed report was evaluated from three dimensions: format correctness, content completeness and readability, and information richness. If the assessment fails, determine the problem type: If the issue is related to content semantics or formatting, a modification suggestion will be sent to the completion agent. If the problem is due to missing or insufficient information, a retrieval request is sent to the analysis agent.

[0043] Existing methods often employ a pattern of indiscriminate feedback to all agents when an evaluation fails, leading to confusion about responsibilities among agents and low optimization efficiency. This invention addresses this by using problem type identification and targeted feedback mechanisms. It ensures that completion agents only handle format or semantic issues, and analysis agents only handle information insufficiency issues, avoiding resource waste and significantly improving iteration efficiency. Three evaluation dimensions cover formal specifications, basic content quality, and content value, forming a comprehensive verification of defect report quality. This prevents missed issues due to missing evaluation dimensions, such as reports with correct format but insufficient information being mistakenly judged as acceptable. By subdividing problem types, it ensures that each problem is handled by a corresponding responsible agent, preventing unattended issues or duplicate processing by multiple agents. This further guarantees that the final report meets developer requirements, such as information richness meeting the technical requirements for defect reproduction and repair.

[0044] In one specific embodiment, the method provided by the present invention consists of four parts: task decoupling, analysis phase, completion phase, and evaluation phase, the overall framework of which is as follows: Figure 2As shown. First, in the task decoupling section, the task of completing and optimizing defect reports is divided into three independent sub-stages: defect analysis, report completion, and quality assessment, each executed by a different large language model agent, thereby effectively reducing the cognitive load on each agent. To clarify the responsibilities of each agent and improve the efficiency of their sub-task completion, a structured prompt template is designed. The template includes received information, team description, and responsibility statement, which helps each agent... The agents accurately understand the task requirements and ensure output quality. To effectively achieve "human expert collaboration," a dynamic feedback and iterative optimization mechanism is introduced, enabling three agents to conduct multiple rounds of cross-validation and collaborative optimization around the same defect report. In specific execution, given a task T, the method of this invention completes the mapping from input to output through multi-agent collaboration, defined as T: X→Y, where T represents the defect report completion and optimization task, X is the original defect report content as input, and Y is the completed and optimized defect report as output. By introducing a "phased multi-agent collaboration" mechanism, the method of this invention achieves efficient collaboration between different agents while ensuring the independence of tasks at each stage, thereby improving the quality of defect reports.

[0045] Task decoupling In the real world, complex tasks often involve multiple stages, roles, and information flows. To complete such tasks efficiently and reliably, human experts typically employ a "phased processing, multi-role collaboration" mechanism. This involves breaking down the task into several sub-tasks with clearly defined boundaries, each assigned to a different expert independently, thereby reducing complexity and improving collaborative efficiency. Inspired by this, the originally highly coupled defect report completion and optimization task process is refined into three consecutive stages: defect analysis → report completion → quality assessment, with a dedicated large language model agent configured for each stage. The analysis agent identifies key missing elements in the defect report; the completion agent supplements necessary technical details and contextual information based on the analysis results; and the assessment agent evaluates and provides feedback on the format, completeness, readability, and information richness of the completed content. This "role solidification" mechanism not only ensures clear task boundaries and division of labor but also effectively reduces the cognitive load and illusion risk of the large language model when handling complex tasks, improving the reliability and accuracy of the generated content. Furthermore, a structured prompt template is designed to help each agent more efficiently understand and perform its role responsibilities. The template consists of three parts: (1) Received Information: Defines the input content and external knowledge that the agent may receive; (2) Team Description: Provides the collaborative environment in which the agent is located, the functional roles of other agents, and the background of complex tasks; (3) Responsibility Description: Clarifies the responsibility boundaries and task list of the current agent, including output content and quality requirements. The introduction of structured prompts enables each agent to have a clear contextual understanding and role positioning when performing sub-tasks, thereby significantly enhancing the overall collaboration efficiency and generation quality. In summary, this invention introduces the concept of "phased, multi-agent collaboration" into the defect report completion and optimization task, providing a scalable and controllable paradigm for the reliable and robust application of large language models in complex software engineering tasks. This not only improves the quality of defect reports but also provides a reference for the implementation of multi-agent collaboration in engineering practice.

[0046] Analysis phase In this invention's method, based on structured prompts, an LLM (Local Management Module) is guided as an analysis agent to perform contextual analysis of the defect report content, focusing on identifying the types of missing fields in the report (such as observed behavior, expected behavior, reproduction steps, etc.). First, the analysis agent focuses on identifying and parsing the key elements of the report content, clarifying the specific category of missing information. Next, the analysis agent retrieves external knowledge related to the current report topic, searching for materials that can provide reference for the missing content, and further integrates this information to generate completion suggestions. Furthermore, the analysis agent may receive feedback and retrieval requests from the evaluation agent, requiring it to perform deeper external knowledge retrieval to form more comprehensive and detailed completion suggestions, supporting the execution of subsequent completion tasks. This stage of the process can be represented as follows:

[0047] Whether a retrieval request is received from the evaluation agent during this process will determine the specific behavior of the analysis agent. If no retrieval request is received, the analysis agent will only respond to structured prompts. Under the guidance of the assessment agent, the current defect report X is analyzed, external knowledge is retrieved, and completion suggestions are generated. Otherwise, the analysis agent will combine the retrieval request, defect report X, and structured prompts received from the assessment agent. Under guidance, additional knowledge was retrieved and completion suggestions were updated. The structured instruction settings of the analysis agent were configured as follows: Figure 3 As shown.

[0048] Specifically, the analysis of the agent's work consists of the following four key steps.

[0049] Analyzing and identifying missing information. Existing research indicates that key fields such as observed behavior, expected behavior, and reproduction steps can help developers understand the cause and location of defects in a timely manner, thereby enabling efficient remediation. Observed behavior provides the actual manifestation of the defect, helping developers understand the current software problem. Expected behavior provides the ideal performance of the software and serves as a standard for problem remediation. Reproduction steps help developers reproduce the problem through specific operations, thus more efficiently locating and fixing defects. Therefore, the analysis agent first performs semantic analysis on the existing content in the report, identifying missing fields from the three dimensions of observed behavior, expected behavior, and reproduction steps.

[0050] Relevant Knowledge Retrieval. Retrieving external knowledge is a crucial step in defect report completion and optimization tasks. It helps the analysis agent retrieve knowledge from historical databases that matches the current report's topic and has similar content, and then generates suggestions for completing missing fields through further integration. Specifically, the analysis agent automatically invokes tools to help it retrieve relevant web links, discussion content, and user answers from the internet, enabling it to quickly obtain relevant contextual information from a large amount of external resources. Furthermore, discussions and issues related to open-source projects are often concentrated on platforms like Bugzilla. Bugzilla provides a powerful RESTful API interface that supports searches by keywords, defect number, product, version, and other methods. The analysis agent can directly use the Bugzilla API to retrieve relevant knowledge. For example, using a Bugzilla query like: `GET / bug?keywords=NaN&keywords=tf.select&product=TensorFlow&status=NEW` will retrieve questions about the `tf.select` function. This query will retrieve all new defect reports containing "NaN" and "tf.select", which may be relevant to the current defect. In this way, the analysis agent can efficiently obtain solutions, discussions, or remediation suggestions related to the current defect. Retrieving similar defect reports is crucial in this process. However, traditional text or embedding similarity matching methods often result in low-quality matches due to missing information, as target defect reports often lack key information. To address this, a hierarchical retrieval and matching framework is proposed. This framework first calculates the similarity of centroids and then compares the similarity of specific statement sets, achieving multi-level optimized matching. Specifically, the three-class centroid learning process in the training phase summarizes the current knowledge topic by aggregating different types of key information (observed behavior, expected behavior, and reproduction steps). This method can cluster each type of statement separately and generate its own centroid. Thus, regardless of which type of information the target defect report lacks (e.g., expected behavior or reproduction steps), the system can still perform effective matching using information from other classes, enhancing adaptability and flexibility to different defect reports. In the matching phase, the system filters candidate top-k centroids to narrow the matching range. Then, it meticulously compares the target statement set with statement sets corresponding to multiple centroids. This matching method can more accurately capture the similarity between the target report and candidate reports, improving the relevance of the search results. The above process is as follows: Figure 4 As shown.

[0051] Generate completion suggestions. Besides receiving similar defect report groups, the agent may retrieve duplicate or irrelevant knowledge. Therefore, the analysis agent needs to further measure the similarity between relevant knowledge fragments and the target task to ensure that the generated suggestions effectively help the completion agent perform the completion task. For example, when retrieving external knowledge to help complete the "reproducibility steps," the agent may retrieve knowledge fragments that are irrelevant or redundant to the task, such as "use tf.select to handle NaN values" or "add a step to check for NaN before the tf.select operation." This knowledge is not very helpful for completing the "reproducibility steps." Therefore, the structured instructions explicitly require the analysis agent to selectively integrate relevant knowledge to ensure that the generated suggestions accurately and effectively support the completion process.

[0052] Receiving Feedback. When the analysis agent receives a request from the evaluation agent to retrieve more knowledge, it treats this specific feedback as a prompt and automatically triggers a dynamic optimization process, i.e., the collaboration between the analysis agent and the completion agent enters a new iteration cycle. At this time, the analysis agent re-executes steps (1)-(3) and adjusts the knowledge retrieval strategy according to the requirements of the evaluation agent to generate more detailed suggestions, thereby helping the completion agent to carry out a new round of missing content generation.

[0053] Completion phase During the defect report completion and optimization process, the completion agent's work is carried out under the collaboration and supervision of the analysis agent and the evaluation agent. This process can be represented as:

[0054] Here, X represents a shared variable. The completion agent takes the input defect report text X, the prompt instruction P, and the completion suggestion as input. analyse Under the guidance of [unspecified entity], the report output is completed after the key fields are filled in. The content in X is updated accordingly. In this stage, the completion agent mainly involves two core responsibilities.

[0055] (1) Generate missing content based on the analysis agent's suggestions. The completion agent receives completion suggestions, including missing fields (e.g., "reproducible steps", "expected behavior", etc.) and details to consider during the completion process, and performs the text generation task using LLM. The generated content typically follows a specific format and structure as required by the instructions to ensure completeness, clarity, and accuracy. Note that the completion agent is only responsible for generating the missing content and does not change the content of other fields.

[0056] (2) Content optimization based on the evaluation agent's suggestions. The completion agent will adjust the generated content based on the feedback from the evaluation agent. This mainly includes two aspects: format and readability. In terms of format, defect reports are required to be output strictly in JSON format and include five parts: number, title, observed behavior, expected behavior, and reproduction steps. This makes the file structure clear and hierarchical, helping developers to quickly understand the content when reviewing the report. Therefore, when the evaluation agent finds that the report's content format is incorrect or does not conform to the preset, it will return the suggestion "The report format has a problem, please reorganize the report content in JSON format." In terms of readability, when the evaluation agent provides suggestions such as "The reproduction steps contain lengthy and repetitive expressions, please reorganize this part to ensure the report is concise," the completion agent will optimize the corresponding part as required.

[0057] The behavior of the completion agent is guided by structural hints to ensure that it performs the aforementioned duties with high quality. The instruction settings are as follows: Figure 5 As shown.

[0058] Evaluation phase At this stage, the assessment agent is responsible for evaluating the quality of defect reports and determining, based on the assessment results, whether to approve the current version or send back feedback recommendations. Its core responsibilities focus on assessing report quality and dynamically selecting appropriate feedback paths. This process can be represented as:

[0059] Among them, the evaluation agent LLM evaluate The output includes two results: pass or suggestion (request). When the report quality fails, the evaluation agent, guided by the structured prompt instruction P, sends a modification suggestion back to the completion agent or a retrieval request back to the analysis agent regarding the defect report content X. Specific structured instruction settings are as follows: Figure 6 As shown, the evaluation agent relies on the following three key technology modules.

[0060] Quality assessment mechanism. The assessment agent uses predefined prompts to guide the LLM in evaluating the quality of the supplementary content from three aspects: report format correctness, content completeness and readability, and information richness, to determine whether it meets the expected standards. Specifically, the report is required to be output in JSON format and include five fields: number, title, observed behavior, expected behavior, and reproduction steps. Completeness helps check whether missing content has been supplemented. Readability and information richness further ensure that the report content can effectively help developers perform defect fixing actions.

[0061] Feedback path triggering mechanism. Based on the type of problem identified during the evaluation process, the evaluation agent automatically selects a feedback path to support subsequent optimization. For example, if the agent determines that there are issues such as semantic ambiguity or errors in the report, it directly sends feedback to the completion agent and provides modification suggestions. If the evaluation results show that the report lacks key information or the information content of the completion is insufficient, it sends a feedback request to the analysis agent, guiding it to re-retrieve relevant background knowledge and generate new completion suggestions, thereby executing a new round of completion process.

[0062] A multi-round review and reassessment mechanism is implemented. The assessment agent supports a multi-round iterative review process. For each revised version submitted by the completion agent, the system will restart the quality assessment process until the defect report of the current version is approved. A maximum number of iterations N is defined. When the maximum number of iterations is reached or the assessment agent confirms that the report quality is acceptable, the workflow terminates and the final defect report content is output.

[0063] Experimental Analysis Research Questions To evaluate the performance of the method of the present invention, the following five research questions were designed.

[0064] Research Question 1: How does the performance of this invention compare to existing defect report completion methods? Specifically, the performance of the method of this invention is evaluated on BLEU, Sentence-BERT, ROUGE-L, and METEOR metrics, comparing it with the best-performing methods of defect report completion and optimization in traditional machine learning paradigms and LLM paradigms.

[0065] Research Question 2: How do different LLMs used as surrogate experts affect the performance of the method in this invention? Large language models with different parameter counts, such as LLaMA, Qwen, and DeepSeek, were selected as surrogate experts to evaluate the impact of different LLMs on the performance of the text method.

[0066] Research Question 3: What are the effects of phased and multi-agent collaboration in this invention? That is, what is the contribution of each agent expert in the process of completing the complex completion and optimization tasks? Research Question 4: What are the effects of the number of iterations and the number of missing fields on the performance of the method in this invention? Set the number of iterations as a hyperparameter and observe its impact on the performance of the text method. Additionally, statistically analyze the performance of the method in this invention for completing different fields across four metrics.

[0067] Research Question 5: Can the hierarchical retrieval and matching method improve the accuracy of knowledge retrieval? Specifically, we will observe the impact of using the hierarchical retrieval and matching method for similar defect report retrieval on performance. Furthermore, we will set the number of retrieved defect reports as the hyperparameter m and observe its value's effect on the completion effect of the method in this invention across four indicators.

[0068] Experimental setup Experimental data Four widely used open-source projects were selected to construct an experimental dataset: GitHub, Eclipse, Mozilla, and Tomcat. Details are shown in Table 1. The four datasets collectively cover the client, system, server, and general open-source ecosystem, encompassing mainstream programming languages ​​such as Java, C, and C++. For example, GitHub covers various application scenarios including tool libraries, front-end frameworks, and DevOps, involving programming languages ​​such as Java, Python, and C++. Eclipse covers editors, compilers, UIs, and toolchain integration, with detailed text specifications and reproduction information. Mozilla, with its strict templates, covers system-level issues such as multi-processing, graphics rendering, and network protocols. Tomcat, primarily based on Java, emphasizes server-side features such as concurrency, resource management, and configuration. To verify the performance of the method, 600 complete defect reports were randomly selected from each project, totaling 2400 defect reports. The observed behavior, expected behavior, and reproduction steps were randomly deleted from each project's defect reports. This deletion strategy involved deleting one or two pieces of information from the observed behavior, expected behavior, and reproduction steps, resulting in each complete defect report corresponding to two incomplete defect report variants. Considering the computational cost of large language models, a stratified random sampling method was adopted, drawing 50% (300 samples) from the incomplete samples of each project, ultimately forming 1200 experimental samples. This experimental setup ensures both data diversity and controlled experimental scale. Since the experimental results need to be compared with real, complete defect reports, feature deletion was used to simulate incomplete reports in real-world scenarios. All data processing was automated to ensure experimental repeatability. Additionally, an extra 1200 complete defect reports were selected to train the hierarchical retrieval and matching method.

[0069] Table 1. Details for each dataset

[0070] Experimental indicators To comprehensively evaluate the quality of defect report completion, a multi-dimensional evaluation index system is adopted, including three levels: surface matching degree, semantic similarity, and text fluency, using four indicators: BLEU, SentenceBERT, ROUGE-L, and METEOR. BLEU (Bilingual Evaluation Understudy): The BLEU metric is widely used in text generation tasks in Natural Language Processing (NLP). It is primarily used to quantify the similarity between automatically generated text and reference text. In the defect report completion task, BLEU assesses this similarity by analyzing the degree of overlap of n-grams between the completed defect report text and the actual defect report text.

[0071] SentenceBERT focuses on evaluating semantic similarity. It measures the semantic closeness of the reported text by mapping the completed defect report text and the actual defect report text to a unified vector space and calculating their cosine similarity. This approach allows for a deeper understanding of the similarity of report content, rather than relying solely on superficial word matching.

[0072] ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation): ROUGE-L captures sentence-level structural similarity by considering the order of words in the generated and actual text. This evaluation method focuses not only on word matching but also on the order in which these words are arranged in the sentence, thus providing a more nuanced understanding of text coherence and relevance. In the defect report completion task, similarity is assessed by comparing the LCS (Low Segment Similarity) between the generated and actual content.

[0073] METEOR (Metric for Evaluation of Translation with Explicit ORdering) is a comprehensive evaluation metric. Compared to traditional evaluation methods such as BLEU, it incorporates linguistic features such as stemming and synonyms, aligns words between the generated and reference summaries, allows for partial matching, and better handles different semantic expressions. Simultaneously, METEOR penalizes word order differences, enhancing the evaluation of output fluency and grammatical correctness. Overall, METEOR is a comprehensive metric that considers multiple factors and aims to more accurately measure the quality of generated summaries. In this invention, it evaluates quality by comparing the generated text with the actual text at the word level.

[0074] Baseline To fully demonstrate the advantages of the method proposed in this invention and to more objectively evaluate the performance of the model, the following three baseline models were selected.

[0075] BEE: Utilizes machine learning technology to automatically check the completeness of defect reports and proactively contacts the report submitter to request missing information.

[0076] ChatBR: It identifies the completeness of defect reports by using a fine-tuned BERT model and uses LLM to autocomplete defect reports by combining prompting learning techniques.

[0077] AdbGPT: By leveraging few-shot learning and chain-like reasoning, it helps LLMs acquire human knowledge and logical reasoning abilities, thereby generating reproducible steps in a developer-like manner.

[0078] Tab: First, train the BERT model to divide the defect report into four components, and then perform a title generation task on the templated report based on a single LLM.

[0079] BRMDS: By integrating the PEFT strategy and a carefully designed instruction dataset, it can generate comprehensive, accurate and easy-to-understand multi-dimensional summaries.

[0080] Large Language Model To verify the universality of the method of this invention for different LLMs, experiments were conducted on LLMs of various sizes. The following is a detailed description of these LLMs.

[0081] LLaMA: LLaMA is a family of large-scale open-source language models developed by Meta, focusing on natural language processing tasks. This model performs exceptionally well in software engineering, assisting in various tasks such as summarization, defect detection, and software design. This invention selects two versions: Llama-3-3B and Llama-3-8B.

[0082] Qwen: Qwen is a Transformer-based language model developed by Alibaba Cloud. It is pre-trained using the task of predicting the next word and is suitable for tasks such as text generation, question answering, dialogue, translation, and text classification. The model offers various parameter sizes from 0.5B to 72B. This invention selects two versions: Qwen-2.5-14B and Qwen-2.5-32B.

[0083] DeepSeek, proposed by DeepSeek Inc., primarily includes text generation, dialogue capabilities, coding, mathematical calculations, and reasoning tasks. It can be integrated into various downstream systems or applications to provide users with intelligent dialogue and content generation services. This invention selects two versions: DeepSeek-R1 and DeepSeek-V3.

[0084] GPT: An autoregressive language model based on Transformer, developed by OpenAI, which uses next-to-token prediction for large-scale pre-training. The model is suitable for text generation, question answering, dialogue, translation, summarization, information extraction, and classification. This invention selects two versions: GPT-4o and GPT-o3.

[0085] Claude: A language model developed by Anthropic, suitable for tasks such as text generation, question answering, and dialogue, with outstanding performance in tasks such as long text comprehension and document and image parsing. This invention selects two versions: Claude-3-Sonnet and Claude-3-Opus.

[0086] Parameter settings For this invention and other methods, access is made using the API of a different LLM, which does not receive updates. In all experiments, the maximum token size was set to 512. For this invention, the number of iterative feedbacks between roles was limited to 4. For parameter settings during the execution of the baseline methods, refer to the results reported in their original papers.

[0087] Research Question 1: How does the performance of this invention compare to existing defect report completion methods? Table 2. Performance comparison of the method of the present invention with other baselines

[0088] Table 2 shows the performance of the proposed method (based on GPT-o3) compared to existing baselines on BLEU, SentenceBERT, ROUGE-L, and METEOR metrics. As can be seen from the table, the proposed method improves performance on the four public datasets by 10.41%, 7.52%, 13.55%, and 16.64% respectively, compared to the second-best performing ChatBR. This indicates that the proposed method can more accurately identify missing information in defect reports, generate text content that is more realistic and of higher quality, and is clearer and more in line with human language habits. The inventors believe that this advantage may stem from task decoupling and role solidification, which help agent experts at different stages complete sub-tasks with high quality. Furthermore, compared to one-time fill machine learning methods and methods based on a single LLM, the proposed method shows significantly superior performance. This advantage arises from the iterative feedback mechanism, which promotes collaboration and supervision among multiple agents, fully leveraging the advantages of collective intelligence to achieve multiple cross-validations and collaborative optimizations of the same defect report, thereby significantly improving report quality. Furthermore, the method of this invention outperforms the AdbGPT method, which employs few-shot learning and chain-like reasoning for defect report completion. It's worth noting that chain-like reasoning is a successful hinting technique that solves reasoning problems by generating intermediate reasoning steps. However, unlike this invention, AdbGPT lacks further review and correction of the generated results when generating defect reports, failing to perform timely quality control and adjustments at each stage, thus limiting its overall performance. In contrast, this invention ensures the quality of the final report through dynamic feedback and iterative optimization mechanisms.

[0089] Research Question 2: How does using different LLMs as proxy experts affect the performance of the method in this invention? Table 3. Impact of different LLMs on the performance of the method of this invention

[0090] To investigate the performance of the model provided in this invention on LLMs of different types and sizes, defect report completion tasks were implemented based on Llama, Qwen, DeepSeek, GPT, and Claude, respectively. The experimental results are shown in Table 3. As can be seen from the table, model performance improves with increasing parameter size, which is reasonable. This is because larger-scale models can better capture details from existing information in defect reports, thereby improving the quality of text generation. Especially in complex natural language processing tasks, larger models can understand and process more information, thus generating more accurate and coherent text. Secondly, among the LLMs used, GPT-o3 was found to perform best in the defect report completion generation task, providing higher quality generation results than other models. For example, on the BLEU and SentenceBERT metrics, GPT-o3 improved performance by 6.93% and 11.07% compared to Llama-3-3B on the four datasets, respectively. This advantage demonstrates GPT-o3's superior ability to generate high-quality text. Compared to GPT-4o, GPT-o3's advantage mainly lies in its ability to handle complex reasoning and logical analysis tasks. In defect report completion tasks, GPT-o3 focuses on collaboration and feedback among multiple agents, continuously optimizing report content. In tasks requiring high collaboration and multiple rounds of feedback, GPT-o3 can generate more relevant completion content through more accurate reasoning and logical analysis. Therefore, GPT-o3 fully leverages its strengths and performs exceptionally well in handling such tasks. Conversely, Claude's focus on stability and security leads to overly conservative and indecisive generation, potentially lacking the depth and flexibility of GPT models in text generation tasks requiring detailed supplementation.

[0091] Research Question 3: What are the effects of phased and multi-agent collaboration in this invention? Since the completion agent is responsible for performing the text generation task for missing fields, ablation experiments were not conducted on the completion agent to ensure the task's successful completion. However, ablation experiments were performed on different agents to evaluate the analysis agent and assess its contribution to the missing field completion process.

[0092] `w / o analyze` indicates that the analysis agent is removed, and defect report completion and optimization tasks are performed solely through the collaboration of the completion agent and the evaluation agent. In this case, the role of the analysis agent is missing, and the completion agent relies on feedback from the evaluation agent to fill in the missing content without receiving any guidance from analysis or external knowledge regarding the existing content of the report.

[0093] `w / o evaluate` indicates that the evaluation agent is removed, and defect report completion and optimization tasks are performed solely through the collaboration of the analysis agent and the completion agent. In this case, the role of the evaluation agent is missing. Although the analysis agent still performs contextual analysis of the report, retrieves external knowledge, and provides completion suggestions, it lacks evaluation and feedback on the quality of the completed content.

[0094] Table 4. Impact of the absence of different agency experts on defect report completion performance

[0095] The experimental results are shown in Table 4. As can be seen from the table, the absence of any role significantly impacts the performance of defect report completion and optimization. Specifically, when the analysis agent is missing, the performance of the generated defect report content on the four metrics decreased by 3.80%, 2.31%, 5.27%, and 5.45%, respectively. This result indicates that the analysis agent plays a crucial role in the completion process, as it is responsible for identifying missing key information from the context and providing detailed completion suggestions. The absence of the analysis agent leads to a lack of proper guidance for the completion task, affecting the completeness and accuracy of the report. When the evaluation agent is missing, the generated defect report content on the four metrics decreased by 6.85%, 5.92%, 9.62%, and 10.77%, respectively. This shows that the evaluation agent plays a vital role in the quality of the completed content, especially during iterative feedback. By checking and optimizing the completed content, the evaluation agent ensures that the generated report meets high-quality standards. The absence of this role disrupts the phased multi-agent collaboration mechanism in the completion process, resulting in a significant decrease in the quality of the completed content. In conclusion, analysis agents and evaluation agents play important roles in the team. The absence of any one of these roles disrupts the collaboration and feedback process among multiple agents, leading to a decline in the quality of the generated missing information. Therefore, in the method of this invention, the roles of the analysis agent and the evaluation agent are indispensable; only when all three work together can the high-quality completion of complex tasks be ensured.

[0096] Research Question 4: What are the effects of the number of iterations and the absence of different fields on the performance of the method in this invention? To investigate the impact of the number of iterations on the performance of the model in this invention, the number of iterations was set as the hyperparameter t, and the effects of t=2, t=4, and t=6 on the performance of the model on four evaluation metrics were observed. The experimental results are as follows: Figures 7 to 10 As shown.

[0097] As shown in the figure, the model performance initially increases and then decreases with the increase of the iteration number t, reaching its best performance at t=4. This indicates that setting a reasonable number of iterations can effectively improve model performance, but excessive multi-agent iterations may lead to performance degradation. This phenomenon may be closely related to the illusion phenomenon of LLM. Although this invention decouples defect report completion and optimization into three independent stages through task decoupling, effectively reducing task complexity and cognitive load on agent experts, the multi-agent collaboration process may exacerbate the propagation of erroneous information. Specifically, with the increase of the iteration number, the model may begin to "overfit" certain error patterns, and even small errors may accumulate and propagate in each iteration, ultimately leading to a decline in the quality of generated content. Therefore, in the operation of this invention, selecting a reasonable number of iterations is particularly important, as it helps to better control the quality of the model-generated content.

[0098] Furthermore, the inventors investigated the performance of the method of this invention on four metrics when completing different missing fields (observed behavior OB, expected behavior EB, and reproduced step S2R). Experimental results are as follows: Figures 11 to 18 As shown.

[0099] from Figures 11 to 18It can be seen that this invention performs best when completing the expected behavior field, but is more difficult when completing the reproduction step field, which is reasonable. Specifically, the expected behavior describes the system behavior expected in the defect report, and is usually closely related to user needs, system design, and functional expectations. Completing the expected behavior usually involves understanding the system functions and inferring common usage patterns. Since the expected behavior is essentially a description of an ideal situation, it can usually be inferred from existing system functional design. For example, if the report describes a failure of a functional module, the completion agent can infer the behavior the system should exhibit when there is no defect, based on the design goals of that functional module. Since the expected behavior usually has relatively standardized expected behavior, the completion process is relatively simple, and this invention performs best in this aspect. In contrast, completing the reproduction steps not only involves detailed operational steps, but also requires consideration of multiple factors such as different environments, configurations, and input data. Especially when there is a lack of direct records and detailed information, completing the reproduction steps is very challenging. For example, the reproduction steps may need to accurately describe the operating system version, hardware configuration, the type and order of specific input data, etc., which are often dynamically changing and often lack clear records. If the reproduction steps in the defect report are not clearly described, the completion task becomes more complex, potentially requiring multiple iterations and more refined reasoning. Nevertheless, this invention, through a phased multi-agent collaboration mechanism, successfully leverages the advantages of teamwork and, to some extent, compensates for the shortcomings in the reproduction step completion process. Even under complex conditions and with insufficient information, the quality of the generated content can be improved through multiple rounds of iterative feedback.

[0100] Research Question 5: What are the effects of hierarchical retrieval matching methods and different hyperparameters m on the performance of the method in this invention? To investigate the impact of the hierarchical retrieval matching method and hyperparameter k on the performance of the model of this invention, the inventors conducted relevant experiments to observe the performance of the generated content on four performance indicators. The experimental results are shown in Table 5. Here, w / omatch-2s represents the experimental results after removing the hierarchical retrieval matching method. For the hyperparameter m, the performance impact of m=2, m=4, and m=6 on the model of this invention on the four evaluation indicators was observed.

[0101] Table 5. Impact of hierarchical retrieval matching method and hyperparameter k on defect report completion performance

[0102] As can be seen from the table, the hierarchical retrieval and matching method proposed in this invention significantly improves the quality of the completed content. Specifically, when the hierarchical retrieval and matching method is not used for similar defect report group retrieval, the performance of the generated defect report content on the four indicators decreased by 3.07%, 3.22%, 3.22%, and 3.48%, respectively. This result indicates that the knowledge provided by the hierarchical matching retrieval method is crucial for the analysis agent to generate completion suggestions. Similar defect reports can help the analysis agent quickly focus among numerous defect reports, understand the cause and manifestation of the current defect, and thus form the optimal completion suggestion. This not only avoids generating irrelevant or inaccurate content but also effectively mitigates the influence of the illusion phenomenon. Furthermore, the impact of the number of similar defect report groups on model performance was further analyzed. As shown in Table 5, when m=4, the performance of the completed text is optimal on the four indicators. The main reason for this phenomenon may be that when m=2, the provided knowledge is insufficient to help the analysis agent form accurate completion suggestions; while when m=6, since the prompt information is input to the analysis agent in text form, the input window size limits the input, resulting in truncation and preventing all retrieved defect reports from being completely input into the LLM. Therefore, selecting an appropriate number of defect reports has a significant impact on the performance of the method of this invention. Nevertheless, the experimental results fully demonstrate the effectiveness of the hierarchical retrieval and matching method.

[0103] Manual assessment To evaluate the completion performance of the method of this invention, the inventors recruited eight experts with software development experience to conduct experiments, including three graduate students and five developers with more than three years of R&D experience. During the experiment, participants completed the experiments independently without communication or discussion. The experiment focused on evaluating the completeness, clarity, and reproducibility of defect reports completed using different methods. Each participant was asked to randomly select 20 defect reports from four publicly available datasets for completion. At the end of the experiment, defect reports completed using different methods were presented, and participants were asked to provide feedback. Participants rated the defect reports using a 5-Likert scale. The experimental results were statistically tested at a predetermined significance level (p<0.05).

[0104] Table 6. Results of manual evaluation

[0105] After comparing the report content completed using different methods, all participants expressed strong support for using the method of this invention in practical work, with average preference scores of 4.24, 4.08, and 4.46 for completeness, clarity, and reproducibility, respectively. To gain a deeper understanding of the effectiveness of the method, the inventors collected feedback from participants and summarized two practical challenges faced when manually completing defect reports. First, understanding the context of existing defect reports is very time-consuming, as it requires grasping the development objectives of different projects, adapting to the non-standardized defect report content written by users with varying cultural backgrounds and skill levels, and analyzing potential operations and components. Second, obtaining feasible suggestions to execute the completion process is not easy, and it is also difficult to determine the correctness of the completed content, resulting in the quality of the final defect report often being closely related to the participant's prior experience and areas of expertise. This problem is particularly prominent for novice completers unfamiliar with the application.

[0106] Discussion of effectiveness and threats This invention employs only widely accepted LLMs (Llama, Qwen, DeepSeek, GPT, and Claude) to study the impact of multi-agent collaboration on the final model performance. While it's impossible to comprehensively evaluate the precise impact of various LLMs on the performance of this invention due to the rapid development of LLMs, the model consistently demonstrates superior defect report completion performance regardless of the LLM-based agent expert. This is because the invention introduces the concept of "human expert collaboration" into defect report completion and optimization tasks, ensuring high-quality completion of each subtask based on a "phased multi-agent collaboration" mechanism, dynamic feedback and iterative optimization mechanisms, and specially designed structured instructions, rather than solely relying on the advantages of LLMs in natural language understanding. Furthermore, the inventors focused solely on the text content of the defect report, neglecting to utilize user-submitted screenshots to aid in report completion. However, while screenshots visually demonstrate the system's execution results when a defect occurs, further analysis of the experimental data shows that the observation behavior in the defect report is essentially a textual description of the screenshot information by the user. Therefore, investing additional time and resources to process screenshot information is unnecessary.

[0107] In addition, the inventors captured 2,400 defect reports from four open-source projects: GitHub, Eclipse, Mozilla, and Tomcat. Most of these reports were presented as unstructured text, with significant variations in text length and quality. Due to the differences in the specific defect report datasets used by other researchers, the inventors cannot assert that the model executed on the current dataset will perform identically on different datasets. However, the inventors provided a detailed description of the experimental setup, which will facilitate the reproduction of the inventors' experiments using more datasets in real-world development.

[0108] Related work Completion of defect report Existing research on improving the quality of defect reports mainly focuses on improving or supplementing key information, such as observed behavior, expected behavior, and reproduction steps. For example, Feng et al. proposed the AdbGPT model, which uses few-shot learning and thought chains to help LLM generate reproduction steps; Zhao et al. proposed ReCDroid+, which improves the program analysis and natural language processing tool Yakusu and automatically reproduces crashes using lexical knowledge in defect reports; Song et al. trained a classification model to identify missing key information and requested necessary information from the submitter through an interactive chatbot system. In addition, research on improving defect report titles has also attracted widespread attention. For example, Chen et al. proposed the iTAPE model to generate report titles based on a sequence-to-sequence framework; Zhang et al. generated defect report titles by fine-tuning the BART model; Rastkar et al. used natural language processing technology to summarize defect reports, thereby generating concise title information. Unlike the above methods, the method proposed in this invention introduces the idea of ​​"human expert collaboration," performing the complex task of defect report completion and optimization in stages, and guiding different agent experts to complete sub-tasks with high quality through structured instructions. This strategy aims to effectively leverage the powerful capabilities of LLM to understand and generate natural language text that meets task requirements.

[0109] Evaluation of defect reports In the open-source community, the quality of defect reports is crucial for developers to locate and fix defects promptly and accurately. Therefore, research on defect report quality assessment has gained increasing attention. Most existing studies employ machine learning (ML) or heuristic rules to classify report quality by extracting various metrics associated with defect reports. For example, Fan et al. extracted features from five dimensions: reporter experience, collaboration network, completeness, readability, and text, and used a random forest classifier to identify valid defect reports. Zimmermann et al. designed CUEZILLA, which evaluates defect report quality by selecting information that defect fixers expect from users as features and training a predictive model using a supervised machine learning algorithm. Unlike CUEZILLA, Vasumathi et al. used four different text classification techniques to build and evaluate a classification model, predicting key features based on historical defect fix knowledge. Furthermore, Chaparro et al. developed DeMIBuD, which combines regular expressions, heuristic rules, natural language processing (NLP), and machine learning (ML) techniques to automatically detect missing observation, expected, and reproduction steps. Unlike the methods described above, this invention uses a dynamic feedback and iterative optimization mechanism to enable multi-agent collaboration in completing defect report completion and optimization tasks.

[0110] Multi-agent collaboration With recent advancements in large language models, multi-agent collaboration has become increasingly prominent in the real world, enabling them to perform complex tasks such as creative writing, reasoning, and decision-making. For example, the MetaGPT framework proposed by Hong et al. introduces metaprogramming techniques, including structured workflows and processes in agent interactions to decompose complex problems. CAMEL provides a role-playing framework where a task-specific agent and two cooperating LLM agents (user and assistant) complete tasks through role-based dialogue. AutoGen allows developers to define flexible agent behaviors and communication patterns, allowing LLM agents to collaborate through dialogue and handle complex tasks by breaking them down into manageable subtasks. Furthermore, self-collaboration proposed by Dong et al. instantiates experts from different domains through cue learning, collaboratively addressing the needs of various stages of code generation tasks. In situations of conflicting goals or limited resources, agents prioritize their personal goals, leading to conflicts or oppositions with others' goals, thus triggering competition among agents. Nevertheless, this competition can still evolve towards collective goals, as seen in scenarios such as debates and strategic games. For example, LLMARENAb is an agent collaboration framework with competition as its primary collaborative type. Benchmarked in seven dynamic game environments, it demonstrates skills such as spatial reasoning, strategic planning, numerical reasoning, risk assessment, communication, adversary modeling, and teamwork. CompeteAI also simulated a game environment where two agents act as restaurant managers competing for 50 customers, illustrating how structured competition drives agents to improve their strategies, consistent with several classic sociological and economic theories. LEGO introduces a multi-agent collaboration framework to generate causal explanations, where competitive collaboration links are predefined. In MetaQA, collaboration between LLM agents occurs early in the training phase, where multiple expert agents are combined and trained together towards a single objective that allows agents to compete for the best candidate answers and identify agents trained on the domain of the input problem. Competitive approaches offer advantages such as promoting robustness, strategic adaptability, and complex problem-solving capabilities within multi-agent systems. However, competition also presents challenges, including potential conflicts that require mechanisms to ensure the competition remains constructive and beneficial to the overall system objectives.

[0111] Summarize In this invention, the inventors propose a method for defect report completion and optimization based on multi-agent collaboration. To achieve this goal, the inventors decouple the complex task into three stages: defect analysis, report completion, and quality assessment, and design independent large language model agents for each stage based on structured instructions. To achieve efficient collaboration among multiple agents, a dynamic feedback and iterative optimization mechanism is introduced, enabling the three agents to perform cross-validation and collaborative optimization around the same defect report. This effectively controls semantic drift, ensuring logical coherence and consistency between the completed content and the facts. Compared with existing methods, the method of this invention can effectively complete the completion task and progressively improve the quality of the completed content. Experimental results show that the method of this invention outperforms state-of-the-art baseline methods on four public datasets.

[0112] Example 2 like Figure 19 As shown, this embodiment provides a defect report completion and optimization system based on multi-agent collaboration, including: The report acquisition module is used to obtain the original defect report; The task decoupling module is used to decouple the task of completing and optimizing the original defect report into three sub-tasks that are executed sequentially: defect analysis, report completion, and quality assessment. The defect analysis module is used to analyze the input raw defect report through the analysis agent, identify the missing key information fields, and generate completion suggestions based on the retrieved external knowledge; The report completion module is used to receive completion suggestions from the defect analysis output through the completion agent, and generate and complete the missing content in the original defect report based on the completion suggestions to obtain the completed report; The quality assessment module is used to evaluate the quality of the completed report through an assessment agent. If the quality assessment fails, feedback information is generated based on the assessment results and sent to the defect analysis subtask and / or report completion subtask to trigger a new round of defect analysis and / or report completion until the assessment passes or the maximum number of iterations is reached.

[0113] Example 3 This embodiment provides a defect report completion and optimization device based on multi-agent collaboration. The device includes a memory and a processor. The memory is used to store computer programs. The processor is used to implement the above-mentioned defect report completion and optimization method based on multi-agent collaboration when executing the computer programs.

[0114] The processor is connected to the memory, and one or more computer programs are stored in the memory. When the electronic device is running, the processor executes one or more computer programs stored in the memory to make the electronic device perform the method of Embodiment 1 described above.

[0115] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0116] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0117] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0118] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0119] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0120] Example 4 In another embodiment of the present invention, a computer-readable storage medium is provided on which a computer program is stored. When the computer program is executed by a processor, it implements the defect report completion and optimization method based on multi-agent collaboration as described above.

[0121] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention according to actual needs. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units can be implemented in hardware or as software functional units.

[0122] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A defect report completion and optimization method based on multi-agent collaboration, characterized in that, include: Obtain the original defect report; The task of completing and optimizing the original defect report is decoupled into three sub-tasks executed sequentially: defect analysis, report completion, and quality assessment. The defect analysis subtask analyzes the input raw defect report through the analysis agent, identifies the missing key information fields, and generates completion suggestions based on the retrieved external knowledge. The report completion subtask receives completion suggestions from the defect analysis output through the completion agent, and generates and completes the missing content in the original defect report based on the completion suggestions to obtain a completed report; The quality assessment subtask evaluates the quality of the completed report through an assessment agent; If the quality assessment passes, a complete and optimized report is output; if the quality assessment fails, feedback information is generated based on the assessment results and sent to the defect analysis subtask and / or the report completion subtask, triggering a new round of defect analysis and / or report completion until the assessment passes or the maximum number of iterations is reached.

2. The method as described in claim 1, characterized in that, The analysis agent, completion agent, and evaluation agent are all implemented using a large language model, and each agent is configured with its own structured prompts, which include the received information, team description, and job description.

3. The method as described in claim 1, characterized in that, The suggestions for generating completion based on retrieved external knowledge include: External knowledge is retrieved using a hierarchical retrieval and matching method, which includes: Calculate the similarity between the target defect report and multiple information centroids pre-stored in the knowledge base, wherein the information centroids are generated based on clustering different types of key information in historical defect reports; Based on the centroid similarity, the Top-K candidate knowledge groups are selected from the knowledge base; Calculate the similarity between the target defect report and the specific statement set in the candidate knowledge group to determine the most relevant target knowledge from the candidate knowledge group.

4. The method as described in claim 3, characterized in that, The information centroid includes the observed behavior centroid, the expected behavior centroid, and the reproducible step centroid.

5. The method as described in claim 1, characterized in that, The analysis agent receives retrieval request feedback from the evaluation agent and adjusts the knowledge retrieval strategy based on the retrieval request feedback to regenerate completion suggestions.

6. The method as described in claim 1, characterized in that, The completion agent receives modification suggestions from the evaluation agent and optimizes the content of the completed report based on the modification suggestions. The optimization includes format standardization and / or content readability.

7. The method as described in claim 1, characterized in that, The evaluation of the quality of the completed report by the evaluation agent specifically includes: The completed report was evaluated from three dimensions: format correctness, content completeness and readability, and information richness. If the assessment fails, determine the problem type: If the issue is related to content semantics or formatting, then a modification suggestion is sent to the completion agent; If the problem is due to missing or insufficient information, a retrieval request is sent to the analysis agent.

8. A defect report completion and optimization system based on multi-agent collaboration, characterized in that, include: The report acquisition module is used to obtain the original defect report; The task decoupling module is used to decouple the task of completing and optimizing the original defect report into three sequentially executed sub-tasks: defect analysis, report completion, and quality assessment. The defect analysis module is used to analyze the input raw defect report through the analysis agent, identify the missing key information fields, and generate completion suggestions based on the retrieved external knowledge; The report completion module is used to receive completion suggestions from the defect analysis output through the completion agent, generate and complete the missing content in the original defect report based on the completion suggestions, and obtain a completed report. The quality assessment module is used to assess the quality of the completed report through an assessment agent; If the quality assessment fails, feedback information is generated based on the assessment results and sent to the defect analysis subtask and / or the report completion subtask to trigger a new round of defect analysis and / or report completion until the assessment passes or the maximum number of iterations is reached.

9. A defect report completion and optimization device based on multi-agent collaboration, characterized in that, The device includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement, when executing the computer program, the defect report completion and optimization method based on multi-agent collaboration as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the defect report completion and optimization method based on multi-agent collaboration as described in any one of claims 1 to 7.

Images