Multi-agent unstructured text review method and system based on delphi method

Abstract

The application discloses a kind of unstructured text review methods and systems based on Delphi method with multi-agent, belong to artificial intelligence and natural language processing technical field, including: to the unstructured document to be reviewed is parsed, structured and block processing, obtains structured document content;According to the document type or content feature of the structured document content, select a group of intelligent agents with different review roles from the multi-agent pool and instantiate, form virtual review expert group;The structured document content is distributed to the virtual review expert group by Delphi method coordinator, arranges anonymous multi-round review process, carries out anonymization summary, feedback and iterative convergence to review opinion, until the end condition of pre-set is satisfied, end review process;Structured review report is generated based on the review process;Ensure that review conclusion is based on objective standard, convert artificial review into automated parallel computing task, substantially shorten organization and execution cycle.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and natural language processing technology, and specifically relates to a method and system for reviewing unstructured text using a multi-agent Delphi method. Background Technology

[0002] Currently, the review of unstructured text mainly relies on human expert meetings or peer review mechanisms. For example, patent CN108664663A designed a server-side collaborative review mechanism that automatically matches reviewers and manages feedback interactions through preset rules, ultimately driving the automated flow of subsequent processes (such as code merging) based on the human approval results. However, this model is not only heavily influenced by cognitive biases such as "groupthink" and the "halo effect," leading to a lack of objectivity in decision-making, but also suffers from a fundamental contradiction between the high human and time costs and the high-frequency iteration requirements of agile development. This often results in reviews becoming mere formalities, making it difficult to achieve the systematic accumulation of tacit knowledge.

[0003] To alleviate the pressure of manual review, the industry has introduced static analysis tools and general document processing technologies based on natural language processing (such as OCR, document classification, and summary generation tools). For example, patent CN118780767A uses knowledge graphs to analyze structural anomalies and generate review suggestions to alleviate the burden on manual reviewers. However, such tools generally suffer from semantic understanding bottlenecks, often limited to document structure extraction or shallow keyword matching, lacking the ability to reason about deep design intentions, complex business logic, and non-functional requirements. Essentially, they only remain at the "formal" level of the document and cannot reach the "substantive logic" of the content.

[0004] Due to current technological limitations, the field of unstructured text review faces a dilemma: while human review can understand complex logic, it is inefficient, costly, and easily influenced by subjective judgment, making it difficult to keep up with the pace of rapid development; while current automated tools, although fast, can mostly only perform simple keyword or format checks and cannot truly understand the business intent behind the document. Summary of the Invention

[0005] To address the aforementioned issues, this application proposes a method and system for automated, multi-round, anonymous review of unstructured text using multi-agent review panels and the Delphi Method. The core idea lies in creatively combining the diverse and deep reasoning capabilities of Large Language Models (LLMs) with the structured, bias-free consensus process of the Delphi Method. By simulating an anonymous, unbiased committee composed of virtual experts from multiple domains, it achieves in-depth, consistent, efficient, and traceable automated review of unstructured text.

[0006] This application provides a multi-agent, Delphi-based method for unstructured text review, including: The unstructured documents to be reviewed are parsed, structured, and segmented to obtain structured document content; Based on the document type or content characteristics of the structured document content, select a group of agents with different review roles from the multi-agent pool and instantiate them to form a virtual review expert group. The Delphi method coordinator orchestrates the entire anonymous multi-round review process, performing anonymized summarization, feedback, and iterative convergence of review comments until the preset termination conditions are met and the review process ends. Generate a structured review report.

[0007] Optionally, the anonymous multi-round review process includes: The review task and the structured document content are distributed to each agent in the virtual review expert group. Each agent independently conducts the first round of review based on its review role and generates the first round of review opinions. The Delphi coordinator collects the review comments from all agents, anonymizes them, and generates an anonymized feedback summary. The anonymized feedback summary is distributed to all agents, and each agent re-examines the content of the structured document based on the anonymized feedback summary and adjusts its own review comments to generate a new round of review comments. Repeatedly perform anonymized aggregation and iterative review until the preset termination conditions are met, at which point the review process ends.

[0008] Optionally, the step of collecting review opinions from all agents through the Delphi coordinator, anonymizing them, and generating anonymized feedback summaries includes: Hides the identifying information that allows the intelligent agent from which the opinion originates; Natural language processing technology is used to perform semantic clustering and summarization of review comments to identify points of consensus, points of disagreement, and unique insights. Generate structured, anonymized feedback summaries.

[0009] Optionally, the semantic clustering and summarization of the review comments includes: Each review comment is converted into a high-dimensional vector using a sentence embedding model. A density clustering algorithm is applied to group semantically similar review comments into one category. Finally, a large language model is invoked to generate a summary title for each category.

[0010] Optionally, the preset termination condition includes: After each iteration, the Delphi coordinator module calculates the current consensus level based on a preset consensus threshold. If the consensus level reaches the consensus threshold or the preset maximum number of iterations is reached, the review process ends.

[0011] Optionally, the consensus threshold may include a statistical dispersion threshold for quantifiable indicators and / or a critical defect recognition rate threshold for qualitative opinions.

[0012] Optionally, the structured review report includes consensus findings, unresolved issues, and information on the review process. The review process traceability information is presented in a visual manner to show the evolution path of viewpoints on key issues in multiple iterations.

[0013] Based on the same inventive concept, this application also provides a multi-agent unstructured text review system based on the Delphi method, comprising: The input processing module is used to parse, structure, and segment unstructured documents to be reviewed, so as to obtain structured document content. The agent orchestration module is used to select a group of agents with different review roles from the multi-agent pool and instantiate them according to the document type or content characteristics of the structured document content to form a virtual review expert group. The Delphi method coordination module is used to orchestrate the entire anonymous multi-round review process through the Delphi method coordinator, and performs anonymization, summarization, feedback and iterative convergence of review opinions until the review process ends when the preset termination conditions are met. The report generation module is used to generate structured review reports.

[0014] Optionally, the system also includes a multi-agent pool containing multiple configurable agent instances driven by a large language model, each agent independently performing review tasks based on its review role.

[0015] Furthermore, this application also provides a computing device, comprising: at least one processor and a memory; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, a multi-agent Delphi-based unstructured text review method as described above is implemented.

[0016] Furthermore, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the above-described method for reviewing unstructured text using a multi-agent Delphi method.

[0017] Compared with the prior art, this application has the following advantages: This application provides a method and system for reviewing unstructured text using a multi-agent Delphi method, comprising: parsing, structuring, and segmenting the unstructured document to be reviewed to obtain structured document content; selecting and instantiating a group of agents with different review roles from a multi-agent pool according to the document type or content characteristics of the structured document content to form a virtual review expert group; orchestrating the entire anonymous multi-round review process through a Delphi method coordinator, performing anonymization summarization, feedback, and iterative convergence of review opinions until the preset termination condition is met to end the review process; and generating a structured review report.

[0018] The beneficial effects of this application are as follows: 1. This application utilizes the Delphi method mechanism to effectively reduce cognitive biases caused by groupthink or halo effects in traditional review by anonymizing the identities and opinions of agents, ensuring that review conclusions are based on objective standards rather than role authority.

[0019] 2. This application transforms manual review into an automated parallel computing task, significantly shortening the organization and execution cycle. This mechanism supports integration into CI / CD pipelines and can serve as an automated quality gate, adapting to the high-frequency iteration requirements of agile development.

[0020] 3. This application overcomes the knowledge blind spots of a single human team by dynamically combining intelligent agents with different technical roles (such as security, performance, compliance, etc.). This multi-role collaborative mechanism enables structured, multi-perspective analysis of documents, effectively reducing the omission of defects.

[0021] 4. The automated process of this application eliminates the impact of human error on review quality, ensuring that the review always follows a consistent logic. The version-based rule base makes the review process standardized, controllable, and reproducible.

[0022] 5. This application solidifies expert experience and industry standards into an external knowledge base, preventing knowledge loss due to personnel turnover. The system can quickly adapt to new standards or new risk models by updating the knowledge base configuration, possessing flexible maintainability and scalability.

[0023] 6. This application not only outputs the final conclusion, but also fully records the review evolution path from disagreement to consensus among intelligent agents. This transparent decision-making record provides an auditable basis for technical decisions and meets the regulatory requirements of high compliance scenarios.

[0024] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 The flowchart illustrating the implementation of the multi-agent Delphi method-based unstructured text review method provided in this application is shown. Figure 2 A schematic diagram of the execution flow of the multi-agent Delphi method-based unstructured text review method provided in this application is shown. Figure 3 The system architecture diagram of the unstructured text review method based on the Delphi method using multi-agent technology provided in this application is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Example 1 This application provides a multi-agent, Delphi-based method for unstructured text review, such as... Figure 1 ,include: The unstructured documents to be reviewed are parsed, structured, and segmented to obtain structured document content; Based on the document type or content characteristics of the structured document content, select a group of agents with different review roles from the multi-agent pool and instantiate them to form a virtual review expert group. The Delphi method coordinator orchestrates the entire anonymous multi-round review process, performing anonymized summarization, feedback, and iterative convergence of review comments until the preset termination conditions are met and the review process ends. Generate a structured review report.

[0029] The specific implementation process of this method embodiment is as follows: Step 1: Parse, structure, and segment the unstructured document to be reviewed to obtain structured document content.

[0030] The system receives unstructured text to be reviewed through a user interface or CI / CD pipeline. The input processing module parses the document, extracts text content, charts, code snippets, etc., and converts them into a unified structured format.

[0031] Step 2: Based on the document type or content characteristics of the structured document, select a group of agents with different review roles from the multi-agent pool and instantiate them to form a virtual review expert group.

[0032] Step 3: Distribute the structured document content to the virtual review expert group through the Delphi method coordinator, arrange the anonymous multi-round review process, perform anonymized summarization, feedback and iterative convergence of review opinions, and end the review process when the preset termination conditions are met.

[0033] Optionally, the anonymous multi-round review process includes: The review task and the structured document content are distributed to each agent in the virtual review expert group. Each agent independently conducts the first round of review based on its review role and generates the first round of review opinions. The Delphi coordinator collects the review comments from all agents, anonymizes them, and generates an anonymized feedback summary. The anonymized feedback summary is distributed to all agents, and each agent re-examines the content of the structured document based on the anonymized feedback summary and adjusts its own review comments to generate a new round of review comments. Repeatedly perform anonymized aggregation and iterative review until the preset termination conditions are met, at which point the review process ends.

[0034] Optionally, the step of collecting review opinions from all agents through the Delphi coordinator, anonymizing them, and generating anonymized feedback summaries includes: Hides the identifying information that allows the intelligent agent from which the opinion originates; Natural language processing technology is used to perform semantic clustering and summarization of review comments to identify points of consensus, points of disagreement, and unique insights. Generate structured, anonymized feedback summaries.

[0035] Optionally, the semantic clustering and summarization of the review comments includes: Each review comment is converted into a high-dimensional vector using a sentence embedding model. A density clustering algorithm is applied to group semantically similar review comments into one category. Finally, a large language model is invoked to generate a summary title for each category.

[0036] Optionally, the preset termination condition includes: After each iteration, the Delphi coordinator module calculates the current consensus level based on a preset consensus threshold. If the consensus level reaches the consensus threshold or the preset maximum number of iterations is reached, the review process ends.

[0037] Optionally, the consensus threshold may include a statistical dispersion threshold for quantifiable indicators and / or a critical defect recognition rate threshold for qualitative opinions.

[0038] Step 4: Generate a structured review report based on the review process described above.

[0039] Once the review process concludes (whether consensus is reached or the maximum number of iterations is reached), the report generation module will activate. It will integrate the results of the final round of reviews, clearly listing all agreed-upon defects and improvement suggestions. For points of disagreement that remain unresolved at the end of the process, the report will also record them accurately and present the main arguments of each party for the final human decision-maker's reference. A key feature is that the report will also include a "Review Evolution" section, which uses charts or text to trace how opinions on key issues (such as a high-risk defect) gradually moved from disagreement to consensus across multiple iterations, thus providing complete transparency and traceability throughout the entire review process.

[0040] Optionally, the structured review report includes consensus findings, unresolved issues, and review process traceability information; wherein, the review process traceability information visually demonstrates the evolution path of viewpoints on key issues in multiple iterations.

[0041] The execution flowchart of the method in this embodiment is as follows: Figure 2 Upon receiving the document to be reviewed, review roles are initialized, and multiple agents conduct the first round of independent review, anonymously submitting their opinions to the Delphi coordinator. The Delphi coordinator summarizes the feedback, generates an anonymized feedback summary, and submits the anonymized feedback summary to multiple agents in the multi-agent pool for subsequent rounds of review. After the multi-agents submit a new round of opinions, the Delphi coordinator summarizes and analyzes them again. When the summary results reach the consensus threshold or the maximum number of iterations, the review process ends, and the final review report is generated.

[0042] This application provides a preferred embodiment of the method, intended to illustrate the complete workflow of this application in a real-world scenario.

[0043] Scenario: A system design document written in Markdown, describing a new user registration and login API, is submitted to this system for review.

[0044] Step 1: Initialization After receiving the document, the input processing module parses its content. The agent orchestration engine identifies the core content of the document through keyword analysis (such as "API", "user", "authentication", "JWT") and decides to instantiate a review panel consisting of three experts from the agent pool: (1) Agent_Security: plays the role of a security expert, whose system tips and knowledge base focus on web security and identity authentication.

[0045] (2) Agent_API_Design: plays the role of an expert in API design and RESTful specifications, and the knowledge base contains best practices for API design.

[0046] (3) Agent_Scalability: plays the role of system scalability and performance expert, focusing on design issues in high-concurrency scenarios.

[0047] Step 2: First Round of Independent Review Three agents analyze the document in parallel and output initial review comments: (1) Agent_Security output: "Flaw category: Authentication mechanism flaw. Severity: High. Description: The login interface / login does not impose any limit on the number of failed password attempts, which makes the system extremely vulnerable to password brute-force and credential stuffing attacks. Recommendation: Implement a rate-limiting and temporary lockout policy based on user account and / or source IP address, for example, lock the account for 15 minutes after 5 failed attempts." (2) Agent_API_Design output: "Defect category: API design specification. Severity: Medium. Description: The API endpoint for retrieving user information is named / getUserInfo, which violates the RESTful design principle that resources should be nouns. In addition, the error response body format defined in the documentation is inconsistent and lacks standardized error codes. Recommendation: Change the endpoint to GET / users / {userId} and define a globally consistent error response structure." (3) Agent_Scalability output: "Dependency category: Security / Performance. Severity: Low. Description: The documentation specifies that the SHA-256 algorithm is used to hash user passwords. While SHA-256 is a secure hash algorithm, its computation speed is very fast, which makes it ineffective against offline brute-force attacks against leaked password databases. Recommendation: Consider replacing it with a computationally intensive password hash function, such as bcrypt or Argon2, to significantly increase the time cost required to crack each password." Step 3: Anonymization Summary The Delphi coordinator module collects the three independent opinions mentioned above, performs anonymization and clustering summaries, and generates the following feedback summary: Summary of the first round of review (1) Identified potential defects: 1) [Security] The login interface lacks protection against brute-force attacks. (Mentioned by 1 / 3 of experts) 2) [Design] API endpoint naming does not conform to RESTful standards, and error response formats are inconsistent. (Mentioned by 1 / 3 of experts) 3) [Security / Performance] The choice of cryptographic hash algorithm needs optimization; computationally intensive algorithms could be considered. (Mentioned by 1 / 3 of experts) (2) Current level of consensus: low. Experts have scattered concerns and have not yet identified any key issues in common.

[0048] Step 4: Second Round of Review All three agents received the aforementioned anonymous summary. After reviewing the opinions of other experts, they reflected on and revised their responses. (1) After seeing the suggestions regarding password hashing algorithms, Agent_Security recognized that this was an important security dimension and updated its opinion: "I completely agree with the views on password hashing algorithms. Using a memory-hard hash function such as Argon2id is the best practice for modern password storage. Correction: Raise the severity of this issue from 'low' to 'high', as it is directly related to the security of user credentials in the event of a data breach." (2) After seeing the security vulnerability in the login interface, Agent_API_Design provided supplementary information from a system design perspective: "I agree that the login interface has serious security risks. Supplementary suggestion: In addition to implementing a locking policy at the application level, a better practice is to configure a global rate limiting policy at the API gateway level. This can not only protect the login interface, but also uniformly protect all sensitive endpoints that may be abused in the future." (3) Agent_Scalability corrected itself after seeing the security experts' emphasis on the severity of hashing algorithms: "After reading the anonymous feedback, I realize that classifying the problem of cryptographic hashing algorithms as a 'low-risk performance' problem is inaccurate. Its core impact is on security. I agree to adjust the severity rating of this problem to 'high' and withdraw my initial rating." This process vividly demonstrates the system's powerful ability as a self-correcting and knowledge-integrating mechanism. Agent_Scalability initially correctly identified the problem from its performance perspective but failed to accurately assess its severity within the overall security framework. Through the anonymous feedback mechanism of the Delphi method, it received input from a more authoritative security perspective and corrected its judgment accordingly. This indicates that the collective intelligence generated by the system transcends the sum of the initial outputs of any single agent, forming a dynamic, deliberative reasoning process.

[0049] Step 5: Consensus Check The coordinator analyzes the feedback from the second round and calculates the degree of consensus: (1) Brute-force attack protection: 3 / 3 agents (100%) agree that this problem is a high-risk flaw. Consensus is reached.

[0050] (2) Upgrade of the password hash algorithm: 3 / 3 agents (100%) agree that this problem is a high-risk defect. Consensus is reached.

[0051] (3) RESTful naming conventions: 2 / 3 of the agents (67%) mentioned or agreed on their importance. The consensus threshold of 80% was not reached, but the disagreement was not significant.

[0052] The coordinator determined that all security issues identified as "high-risk" had reached a high degree of consensus, thus meeting the core review exit criteria.

[0053] Step 6: Generate Report The system process concludes, and the report generation module produces the final review report. The report's "Consensus Discovery" section clearly lists two high-risk security vulnerabilities that must be fixed, along with suggestions for improving the API design specifications. In the "Review Process Retrospective" section, the report uses a table to clearly illustrate the evolution of the "cryptographic hash algorithm" problem, from a "low-risk performance suggestion" proposed by one agent to a "high-risk security consensus" unanimously confirmed by all agents in the second round, providing a solid and credible basis for the final decision.

[0054] Example 2 Based on the same inventive concept, this application also provides a multi-agent unstructured text review system based on the Delphi method, such as... Figure 3 ,include: The input processing module is used to parse, structure, and segment unstructured documents to be reviewed, so as to obtain structured document content. The intelligent agent orchestration engine is used to select a group of intelligent agents with different review roles from a multi-agent pool and instantiate them according to the document type or content characteristics of the structured document content, form a virtual review expert group, and drive the review workflow. The Delphi method coordination module is used to orchestrate the entire anonymous multi-round review process through the Delphi method coordinator, and performs anonymization, summarization, feedback and iterative convergence of review opinions until the review process ends when the preset termination conditions are met. The report generation module is used to generate structured review reports.

[0055] Optionally, the system also includes a multi-agent pool containing multiple configurable agent instances driven by a large language model, each agent independently performing review tasks based on its review role.

[0056] System architecture diagram as follows Figure 3 After the input processing module processes the unstructured documents to be reviewed, it sends them to the agent orchestration engine. The agent orchestration engine distributes review tasks to the multi-agent pool, driving the Delphi method coordination module to conduct the review. The multi-agent pool submits review opinions to the Delphi method coordination module based on the review tasks executed by multiple agents. The Delphi method coordination module summarizes and feeds back to the multi-agent pool for review again, iterates and converges, and obtains the final review result. The final review result is then handed over to the report generation module to generate a structured review report.

[0057] Specifically, the input processing module is the system's data entry point, supporting the reception of unstructured text in various formats, such as Microsoft Word (.docx), Markdown (.md), and PDF (.pdf). Internally, it can integrate various open-source or commercial document parsing libraries; for example, it can use the python-docx library to process Word documents and pandoc or Mistune to process Markdown documents. For PDF documents containing scanned images, this module calls an Optical Character Recognition (OCR) service (which can be integrated with cloud services such as Google Document AI or Azure AI Document Intelligence) to convert them into plain text. After parsing, all document content is uniformly converted into an internal structured data format (e.g., JSON), which preserves the original hierarchical structure (such as chapters, headings, lists, and code blocks). For longer documents, this module also performs intelligent semantic chunking, dividing the document into logically coherent, appropriately sized text blocks, which is crucial for subsequent input into large language models with context-length constraints.

[0058] Specifically, the agent orchestration engine is the "brain" and "command center" of the entire automated review workflow. It is responsible for the lifecycle management of the entire process, from initiation and execution to termination. Its implementation can be based on existing mature multi-agent frameworks, such as Microsoft's open-source AutoGen framework, which provides the underlying capabilities for building agents capable of dialogue and collaboration to complete tasks. In this invention, the orchestration engine defines a specific interaction mode, namely the Delphi mode. It instantiates a DelphiFacilitatorAgent (an implementation of the Delphi coordinator module) as the "host" and multiple ReviewerAgents (instances in the multi-agent pool) as "experts." The engine is responsible for distributing tasks (i.e., documents to be reviewed) to the ReviewerAgents. After each round, it instructs the DelphiFacilitatorAgent to collect, process, and distribute anonymous summaries, and determines the next action based on its returned status (such as "continue iteration" or "reach consensus"), thereby precisely controlling the progress of the entire Delphi process.

[0059] Specifically, the multi-agent pool is the core unit for performing actual review work.

[0060] (1) Agent Instantiation and Diversity: Each agent in the pool is an independent software object instance, the core of which is a client that interacts with one or more large language models (LLMs). The system can be configured with different agent instances using different providers (such as OpenAI's GPT-4, Anthropic's Claude 3, and Google's Gemini) or different versions of LLMs. This heterogeneous design aims to introduce cognitive diversity, simulating members with different backgrounds and ways of thinking in a team of human experts, thereby effectively avoiding the systematic bias that may exist in a single model and improving the robustness of the review results.

[0061] (2) Implementation of the Review Persona: The assignment of the persona is achieved through a carefully designed System Prompt. This is a set of instructions sent before the interaction with the LLM begins, used to set the behavior pattern and knowledge domain of the LLM. For example, the System Prompt for a "Security Expert" agent might be as follows: "You will be playing the role of a top cybersecurity architect with over 15 years of experience. Your knowledge base is based on core industry standards such as OWASP Top 10, CWE, and CIS Benchmarks. In the following task, you will review a software design document. Your sole objective is to identify all potential security vulnerabilities from an attacker's perspective, including but not limited to: injection vulnerabilities, invalid authentication and session management, cross-site scripting (XSS), insecure deserialization, security misconfiguration, sensitive data leakage, etc. For each discovery, you must provide: 1) a vulnerability description; 2) potential impact; 3) a severity score (1-10) under the CVSS 3.1 standard; and 4) specific remediation recommendations. Your output must be in structured JSON format." (3) Integration with the Knowledge Base: To overcome the inherent knowledge deadline problem of LLM and inject the latest domain knowledge, the agent employs Retrieval-Augmented Generation (RAG) technology. When an agent with a specific role (such as a "database performance expert") begins its work, it first uses key concepts in the document (such as "sharding strategy" and "index design") as queries to retrieve relevant best practices, case studies, and anti-patterns from a dedicated, real-time updated database performance optimization knowledge base. This latest information, along with the original document content and system hints, is provided to the LLM as contextual information. This allows the agent's review opinions to be based not only on the general reasoning capabilities of the LLM but also on cutting-edge, highly specialized domain knowledge.

[0062] Specifically, the Delphi coordination module: This module is the core technology for automating the Delphi process.

[0063] (1) Feedback Clustering and Summarization Algorithm: After the coordinator collects review feedback in natural language form from each agent, it performs a multi-stage processing flow. First, a sentence embedding model (such as Sentence-BERT) is used to convert each individual review comment (e.g., "The login interface has no password retry limit") into a high-dimensional vector. Then, a density clustering algorithm (such as DBSCAN) is applied to automatically group semantically similar comments into one category in the vector space. For example, Agent_A's "Lack of brute-force attack protection" and Agent_B's "No limit on the number of login attempts" are highly related semantically, and their vectors will be very close in space, so they are clustered together. Finally, the coordinator takes all the original comment texts in each cluster as a set, calls an LLM, and uses a specific prompt (such as "Please generate a concise and accurate title for the following description of the software defect") to generate a summary title for the cluster, such as "Defect: The login interface lacks brute-force attack protection mechanism".

[0064] (2) Consensus Measurement Algorithm: After each iteration, the coordinator needs to quantify the current consensus level. For quantifiable metrics, such as defect severity scores (assuming a range of 1-5), the coordinator calculates statistical metrics for all agents' scores of the same defect, such as standard deviation or interquartile range. A preset consensus threshold might be "standard deviation less than 0.5". For qualitative opinions (i.e., whether a defect exists), the coordinator calculates the "acceptance rate" of critical defects, which is the percentage of agents pointing out the same type of defect out of the total number of agents in the review group. The consensus threshold can be set to "for defects with a severity of 'high', the acceptance rate must be greater than or equal to 80%". Only when all relevant metrics meet the preset threshold will the coordinator report "consensus reached" to the orchestration engine.

[0065] Specifically, the report generation module is responsible for presenting the final review results to human users in a clear and actionable manner. The generated report typically includes the following sections: 1) Summary: Summarizing the number of key issues found, their severity distribution, and the final review conclusion (e.g., "Approved after modification"). 2) Consensus Findings: Detailing all agreed-upon defects and improvement suggestions, each entry including description, impact, severity, and remediation plan. 3) Outstanding Issues: If the review ended due to reaching the maximum number of iterations, this section lists issues where significant disagreements remain, presenting the main viewpoints and arguments of each party side-by-side. 4) Review Process Retrospective: This is a unique feature of the report, visually demonstrating (e.g., through tables or charts) the "evolution of viewpoints" of one or two key issues across multiple rounds of review, clearly revealing how the system moved from disagreement to consensus through iteration and feedback. This transparency greatly enhances user trust in the review results.

[0066] Example 3 Based on the same inventive concept, this application also provides an electronic device. The electronic device of this application includes at least one processor and at least one storage medium electrically connected to the processor. The storage medium is electrically connected to the processor, wherein the storage medium stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0067] Example 4 Based on the same inventive concept, this application also provides a storage medium storing instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the method described above.

[0068] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-agent method for reviewing unstructured text based on the Delphi method, characterized in that, include: The unstructured documents to be reviewed are parsed, structured, and segmented to obtain structured document content; Based on the document type or content characteristics of the structured document content, select a group of agents with different review roles from the multi-agent pool and instantiate them to form a virtual review expert group. The structured document content is distributed to the virtual review expert group through the Delphi method coordinator, and an anonymous multi-round review process is arranged. The review opinions are anonymized, summarized, fed back, and iteratively converged until the preset termination conditions are met and the review process ends. A structured review report is generated based on the review process described above.

2. The method according to claim 1, characterized in that, The anonymous multi-round review process includes: The review task and the structured document content are distributed to each agent in the virtual review expert group. Each agent independently conducts the first round of review based on its review role and generates the first round of review opinions. The Delphi coordinator collects the review comments from all agents, anonymizes them, and generates an anonymized feedback summary. The anonymized feedback summary is distributed to all agents, and each agent re-examines the content of the structured document based on the anonymized feedback summary and adjusts its own review comments to generate a new round of review comments. Repeatedly perform anonymized aggregation and iterative review until the preset termination conditions are met, at which point the review process ends.

3. The method according to claim 2, characterized in that, The process of collecting review opinions from all agents through the Delphi coordinator, anonymizing them, and generating anonymized feedback summaries includes: Hides the identifying information that allows the intelligent agent from which the opinion originates; Natural language processing technology is used to perform semantic clustering and summarization of review comments to identify points of consensus, points of disagreement, and unique insights. Generate structured, anonymized feedback summaries.

4. The method according to claim 3, characterized in that, The semantic clustering and summarization of the review comments includes: Each review comment is converted into a high-dimensional vector using a sentence embedding model. A density clustering algorithm is applied to group semantically similar review comments into one category. Finally, a large language model is invoked to generate a summary title for each category.

5. The method according to claim 1, characterized in that, The preset termination conditions include: After each iteration, the Delphi coordinator module calculates the current consensus level based on a preset consensus threshold. If the consensus level reaches the consensus threshold or the preset maximum number of iterations is reached, the review process ends.

6. The method according to claim 5, characterized in that, The consensus thresholds include a statistical dispersion threshold for quantifiable indicators and / or a critical defect acceptance rate threshold for qualitative opinions.

7. The method according to claim 1, characterized in that, The structured review report includes consensus findings, unresolved issues, and information on the review process. The review process traceability information is presented in a visual manner to show the evolution path of viewpoints on key issues in multiple iterations.

8. A multi-agent unstructured text review system based on the Delphi method, characterized in that, include: The input processing module is used to parse, structure, and segment unstructured documents to be reviewed, so as to obtain structured document content. The agent orchestration module is used to select a group of agents with different review roles from the multi-agent pool and instantiate them according to the document type or content characteristics of the structured document content to form a virtual review expert group. The Delphi method coordination module is used to orchestrate the entire anonymous multi-round review process through the Delphi method coordinator, and performs anonymization, summarization, feedback and iterative convergence of review opinions until the review process ends when the preset termination conditions are met. The report generation module is used to generate structured review reports.

9. The system according to claim 8, characterized in that, The system also includes a multi-agent pool containing multiple configurable agent instances driven by a large language model, each agent independently performing review tasks based on its review role.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-agent Delphi-based unstructured text review method according to any one of claims 1-7.

11. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the unstructured text review method based on the Delphi method using multiple agents as described in any one of claims 1-7.

Images