Medical image analysis method and system based on multi-agent visual-linguistic debate

Abstract

This invention provides a medical image analysis method and system based on multi-agent visual-language debate, relating to the fields of artificial intelligence and medical information technology. The method first utilizes multiple prompt-driven multimodal large-scale intelligent agents to independently generate multiple different initial image descriptions. Then, through an iterative debate process with structured memory coordinated by a large language model, these initial image descriptions are cross-validated and fused to suppress illusions, strengthen consensus, and ultimately extract high-quality consensus-based semantic descriptions. Finally, through a learnable multi-level visual-semantic fusion analysis model, this semantic description is dynamically injected into the visual feature extraction process, guiding the model to focus on key features, thereby improving the accuracy and robustness of image analysis. This method does not require task fine-tuning of the multimodal large-scale model, avoids dependence on labeled data, and has a high degree of automation and task versatility.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical information technology, and in particular to a medical image analysis method and system based on multi-agent visual-language debate. Background Technology

[0002] Currently, medical imaging diagnosis plays a crucial role in healthcare, particularly in the early detection and accurate identification of complex diseases. Deep learning methods, such as convolutional neural networks (CNNs) and visual transformers (ViTs), have achieved promising results in automated diagnosis by extracting disease-related visual features from medical images. For example, frameworks based on CNNs and visual transformers have been applied to predict the molecular state of tumors from magnetic resonance imaging (MRI) scans, classify breast cancer in mammography, and identify chest diseases from X-ray scans. However, these deep learning models often lack sufficient clinical prior knowledge and a deep semantic understanding of anatomical structures and abnormalities, which limits their diagnostic accuracy.

[0003] To bridge this semantic gap, multimodal large language models have become a promising research direction. These models, trained on massive amounts of general-domain image-text pairs, are capable of complex reasoning and exhibit strong zero-shot generalization ability. Furthermore, their training data includes medical imaging data, giving them a certain level of medical knowledge. Multimodal large language models can provide useful image interpretation and assist in the extraction of diagnostic features. For example, in chest X-ray analysis, multimodal large language models can identify subtle pulmonary edema and generate detailed textual descriptions of its size and location, providing assistance to radiologists. However, directly applying multimodal large language models to medical image diagnosis still presents challenges. Although they possess general medical knowledge relevant to various diagnostic tasks, this knowledge may not be specific enough to fully address practical problems. Multimodal large language models are prone to generating clinically reasonable but actually incorrect inferences, leading to erroneous diagnostic decisions. Furthermore, without large-scale labeled training data, fine-tuning these resource-intensive, multimodal, large-scale language models for specific tasks is extremely challenging, and the fine-tuned models may face serious overfitting problems and have poor generalization ability to test data.

[0004] Therefore, existing medical imaging diagnostic technologies urgently need a framework that can integrate knowledge from multiple models, suppress hallucinations, and achieve deep visual-language fusion to improve the robustness and accuracy of diagnosis. Summary of the Invention

[0005] The purpose of this invention is to provide a medical image analysis method and system based on multi-agent visual-language debate, so as to solve one or more technical problems existing in the prior art.

[0006] On one hand, this invention provides a medical image analysis method based on multi-agent visual-language debate, comprising: Step 1: Obtain the medical image to be analyzed, the first prompt word generation instruction, the target disease name, and the second prompt word; Step 2: Generate a first prompt word for the target disease based on the first prompt word generation instruction and the type of medical image to be analyzed; Step 3: Using the medical image to be analyzed and the first prompt word as input, consult multiple independent agents to obtain multiple initial image description texts; Step 4: Based on the second prompt word, the multiple agents are coordinated through the mediation model to conduct iterative debates starting from the multiple initial image description texts, resolve conflicts and reach consensus, and finally generate a unified image description text. Step 5: Perform feature extraction and feature fusion analysis on the unified image description text and the medical image to be analyzed, and output the disease analysis results of the medical image to be analyzed; The intelligent agent is a multimodal large-scale language model, and the mediation model is a large-scale language model.

[0007] On the other hand, the present invention provides a medical image analysis system based on multi-agent visual-language debate, comprising: The data acquisition module is used to acquire the medical image to be analyzed, the first prompt word generation instruction, the target disease name, and the second prompt word; The first prompt word generation module is used to generate a first prompt word for the target disease based on the first prompt word generation instruction and the type of medical image to be analyzed. The multi-agent module is used to consult multiple independent agents with the medical image to be analyzed and the first prompt word as input, obtain multiple initial image description texts, and respond to the query instructions generated by the mediation model in the iterative debate process; The iterative debate and consensus generation module is used to coordinate the multiple agents through a mediation model based on the second prompt word, to conduct iterative debates starting from the multiple initial image description texts, resolve conflicts and reach consensus, and finally generate a unified image description text. The visual-language fusion analysis module is used to extract and fuse features from the unified image description text and the medical image to be analyzed, and output the disease analysis results of the medical image to be analyzed. The intelligent agent is a multimodal large-scale language model, and the mediation model is a large-scale language model.

[0008] By adopting the above technical solution, the present invention has the following beneficial effects: Through a multi-agent debate process, different model agents are encouraged to verify each other, correct errors, integrate consensus, and resolve conflicts. This can effectively suppress fictional or inaccurate descriptions that may arise in the output of large multimodal language models. At the same time, it strengthens consensus content that reflects correct image understanding, generates higher-quality consensus semantic guidance, and uses this semantic guidance to improve the accuracy and reliability of medical image analysis in the final decision-making stage.

[0009] The entire MAVLD framework leverages the zero-shot or few-shot reasoning capabilities of large multimodal language models, extracting knowledge directly from pre-trained models through automated prompt word generation and iterative debate mechanisms. This eliminates the need for task-specific fine-tuning of these large-scale foundational models, thereby freeing it from dependence on large-scale, high-quality labeled data and avoiding the resource-intensive full-parameter fine-tuning process and the potential risk of overfitting.

[0010] By using a learnable visual-semantic fusion analysis model, high-quality, structured semantic descriptions generated from debates are dynamically injected into the visual feature extraction process through hierarchical alignment and cross-attention. This guides the network model to focus on key image features relevant to the task, effectively guiding and enhancing visual perception with semantic information.

[0011] From the automatic generation of prompts and the coordination of multi-agent debates to the final semantic-visual fusion, the entire framework process is highly automated, reducing human intervention. Furthermore, its core design is not limited to specific disease types or image types, exhibiting good task versatility and architectural scalability.

[0012] The comprehensive validation results on a multi-agent visual-language debate framework on public datasets demonstrate that the proposed method significantly outperforms existing image classification models and fine-tuned multimodal large-scale language models. Attached Figure Description

[0013] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0014] Figure 1 A schematic diagram of the medical image analysis method based on multi-agent visual-language debate provided in an embodiment of the present invention; Figure 2A schematic diagram of the three-stage structure of the multi-agent visual-language debate framework provided in an embodiment of the present invention; Figure 3 This is a comparison example between the unified image description text generated by the method of this invention and the image report given by an image expert. Detailed Implementation

[0015] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] The present invention will be further explained below with reference to specific embodiments.

[0017] The overall architecture of the Multi Agent Vision-Language Debate (MAVLD) framework is as follows: Figure 2 As shown, given a medical image to be analyzed, the multi-agent visual-language debate framework does not directly apply an image classification model, but instead generates relevant image descriptions through a reasoning framework to guide the image analysis process. For example... Figure 2 As shown in b, the multi-agent visual-language debate framework comprises three main phases. In Phase 1, based on automatically generated prompts, multiple multimodal large-scale language model agents are consulted to independently generate disease-related image descriptions. Phase 2 designs a multi-agent debate process where the multimodal large-scale language model agents reach consensus and resolve conflicts to correct potential inaccuracies or fabrications. The debate iterates among the agents, with the large-scale language model mediating a final unified image description based on the complete debate history. Phase 3 is a learnable visual-language fusion module with a multi-level knowledge injection mechanism. This module utilizes the semantic text embedding of the summarized description to guide visual feature extraction for disease classification.

[0018] The following sections will detail the design and implementation of the multi-agent visual-language debate framework through several embodiments.

[0019] Example 1 like Figure 1 As shown, this embodiment provides a medical image analysis method based on multi-agent visual-language debate, including: Step 1: Obtain the medical image to be analyzed, the first prompt word generation instruction, the target disease name, and the second prompt word; Step 2: Generate a first prompt word for the target disease based on the first prompt word generation instruction and the type of medical image to be analyzed; Step 3: Using the medical image to be analyzed and the first prompt word as input, consult multiple independent agents to obtain multiple initial image description texts; Step 4: Based on the second prompt word, the multiple agents are coordinated through the mediation model to conduct iterative debates starting from the multiple initial image description texts, resolve conflicts and reach consensus, and finally generate a unified image description text. Step 5: Perform feature extraction and feature fusion analysis on the unified image description text and the medical image to be analyzed, and output the disease analysis results of the medical image to be analyzed; The intelligent agent is a multimodal large-scale language model, and the mediation model is a large-scale language model.

[0020] Furthermore, the first prompt word generation instruction includes the agent role positioning, task objective, generated content specifications, output data format requirements, and target disease name placeholders, which are used to guide the large language model to generate the first prompt word for the medical image type and the target disease.

[0021] Since medical images contain both disease-related and irrelevant information, it is necessary to guide a multimodal large-scale language model to generate image interpretations relevant to the target disease through appropriate cue words, so that the model's output approximates the professional interpretation of medical imaging experts. To achieve fully automated data processing, this invention designs a first cue word generation instruction. For example, as shown in Table 3, the first cue word generation instruction involves filling in the target disease name in the placeholder position of the target disease name in the instruction, inputting the instruction into a large-scale language model, such as DeepSeek, to generate the first cue word. As shown in Table 6.

[0022] Furthermore, the first prompt word is used to guide each agent to analyze the target medical image and the target disease, and output medical image description text with clinical diagnostic significance.

[0023] Furthermore, the medical image description text comprehensively describes the lesion through multiple specific imaging features, including all observable, measurable, and qualitative imaging indicators, and provides analytical results with medical diagnostic significance. Different types of medical images, such as MRI, CT, and X-ray images, have different imaging features. As shown in Table 6, the imaging features for MRI images include location and relationship, size, signal characteristics, boundaries and special signs, perilesional edema, mass effect, and secondary changes, providing a systematic and structured description of the lesion from multiple dimensions.

[0024] Since multimodal large language models may generate inaccurate image descriptions and fictional content, this invention does not rely on a single multimodal large language model, but instead utilizes the generated first cue word to consult multiple multimodal large language models simultaneously.

[0025] Considering the diversity of multimodal large-scale language models and the balance of computational resources, this embodiment sets up three multimodal large-scale language model agents. In specific implementation, a multimodal large-scale language model with strong image understanding capabilities and potential medical imaging knowledge from existing technologies is preferred.

[0026] For example, such as Figure 2 The system architecture diagram illustrates that, in the current technology, MedGemma, Lingshu, and Chiron-o1 are selected as the three intelligent agents (A, B, and C) participating in the debate. Each multimodal large-scale language model intelligent agent... First prompt Under the constraints, it can independently process the input medical images to be analyzed. Generate initial image description text The outputs of all multimodal large language models constitute the initial image description text set. This set will serve as the starting point for multi-agent iterative debate and reasoning.

[0027] Since different multimodal large-scale language models may generate image description texts for the same medical image to be analyzed, specifically manifested as conflicting opinions on one or more specific image features, this invention proposes a multi-agent debate process to integrate the image description texts output by different multimodal large-scale language models. This process leverages the powerful reasoning capabilities of large-scale language models to resolve conflicts and improve the realism of image descriptions. The debate is iteratively conducted among the agents of the multimodal large-scale language models and coordinated by a dedicated mediator model, preferably a large-scale language model such as DeepSeek. The iterative debate proceeds under the guidance of a second cue word until the debate concludes, ultimately generating a unified and definitive image description text.

[0028] Furthermore, the second prompt includes the mediator model's role positioning, task objectives, authority boundaries, debate steps, debate termination conditions, and debate output data requirements. This guides the mediator model to iteratively debate multiple initial image description texts and output a unified image description text. On the one hand, the second prompt endows the mediator model with professional authority and decision-making power through role positioning, enabling it to make autonomous decisions within its professional responsibilities and authority boundaries. On the other hand, the task objectives and various constraints prevent the mediator model from acting arbitrarily and deviating from the output objective.

[0029] Furthermore, the second cue word restricts the debate steps for each round of debate, specifically including: Step a1: Organize and merge the facts that all agents (doctors) have reached a consensus on according to clinical themes to ensure that the consensus is structured and enriched and to avoid fragmentation; Step a2: Identify unresolved conflicts, name the conflict topics, and record the responses (opinions) of each agent (doctor) to prevent overlooking key disagreements and provide a basis for subsequent focus; Step a3: Prioritize conflicts to identify those that have the greatest impact on the final analysis results, thus avoiding wasting debate rounds on non-critical issues; Step a4 involves strategic questioning and guidance, designing the next query instruction for each key conflict, identifying the target query object, preventing repetitive and inefficient discussions, ensuring information gain in each round of discussion, and promoting convergence. Step a5: Decision-making and controlled convergence. When conflicts cannot be resolved, the majority opinion is adopted. Uncertainties and reasons are recorded to prevent infinite loops and ensure that the optimal feasible solution is reached within a finite number of rounds.

[0030] Furthermore, the conditions for terminating the debate include: reaching a preset number of debate rounds, achieving a strong consensus on all key clinical issues, and having no new information available.

[0031] Furthermore, the debate output data is required to include a specified data format and data structure definition. The data structure definition includes the consensus facts generated in this round of debate, the conflicts to be resolved, the target intelligent agent for the next round of query, and the query instructions.

[0032] For example, the parsing of the second prompt words exemplified in Table 7, as shown in Table 1, includes the following information: Table 1. Examples and Analysis of Secondary Hints

[0033] Furthermore, the iterative debate follows a predefined control flow and maintains relevant information during the debate process, including a debate log and structured memory. The debate log is used to accumulate and store all query commands and their responses given to the multimodal large language model agent; the structured memory is used to store the consensus fact set and unverified active conflicts (i.e., unresolved conflicts) generated in each round of debate; the debate log and structured memory are updated after each round of debate. As shown in Table 2, this embodiment includes examples of data structure definitions used in specific implementations.

[0034] Table 2 Examples of Data Structure Definitions

[0035] Furthermore, the process control of the iterative debate includes: Step b1, initialize the debate log and structure memory: , , in, , , This indicates the initial debate log. Represents the initial structured memory. This represents the initial image description set. Indicates the first cue word, Indicates the first An intelligent agent. The index number representing the intelligent agent. This represents the total number of intelligent agents. Indicates the first The initial image description text generated by the agent. This indicates a medical image to be analyzed.

[0036] Step b2, for the first In each round of debate, the mediation model updates the consensus fact set and the active conflicts to be verified based on the debate logs from the previous round, structured memory, and preset second cue words, and generates query instructions for specific conflicts to be issued to the target agent, expressed by the formula: , , in, Representing the mediation model, Indicates the second cue word, This indicates an updated debate log following the previous round of debates. This indicates an updated structured memory following the previous round of debate. This represents the structured memory updated after the current round of debate. Indicates the first The set of consensus facts generated by the rounds of debate Indicates the first The unverified active conflicts generated by the rounds of debate Indicates the first The query instruction generated by the round of debate is a query that requires the target agent to resolve the current conflict. Indicates the current round of debate. It is a natural number greater than 0.

[0037] Furthermore, the set of consensus facts stored in structured memory refers to the logically or biologically consistent image representation features or combinations of image representation features contained in the image text descriptions generated by multiple agents; the unverified active conflicts stored in structured memory refer to the image representation features or combinations of image representation features contained in the image text descriptions generated by multiple agents that are logically or biologically unlikely to occur simultaneously.

[0038] For example, a combination of imaging features is “well-defined boundaries” + “significant and uniform enhancement”, which usually points to extracranial tumors such as meningiomas. The features are synergistic and consistent, and this combination of imaging features will be stored in a consensus fact set.

[0039] For example, if a lesion is described as “significant cerebral edema” in one imaging description and as “no mass effect” in another imaging description, these two features are contradictory in conventional pathophysiology (significant vasogenic edema usually leads to a mass effect). Such a combination is considered a “mutually exclusive feature” and this combination of imaging features will be stored in the active conflict to be verified.

[0040] Guided and constrained by the second cue word, the mediation model analyzes the debate logs and structured memories stored in the previous round, automatically identifies consistent image performance features or combinations (i.e. consensus fact set) and mutually exclusive image performance features or combinations (i.e. active conflicts to be verified), updates the structured memories, and generates new query instructions based on the active conflicts to be verified, which are used to query one or more target agents that have generated conflicts in the next round of debate.

[0041] Step b3: The target agent re-analyzes the medical image to be analyzed according to the query instruction and generates a new image description, expressed by the formula: , in, Indicates the first The agent in the th... The wheel's response, i.e., the new image description text.

[0042] Each agent re-analyzes the medical images based on the new query instructions, the first prompt words, and the historical debate logs, generating new medical image descriptions. In the new medical image descriptions, agents can choose to retain or modify their viewpoints from the previous round of debate.

[0043] Step b4: Update the debate log for the current round. Specifically, based on the previous round's debate log, append the storage of new query commands generated in the new round of debate and the target agent's response to the new query commands, expressed by the formula: , in, This indicates that all agents are in the 1st... A collection of image description texts obtained after rounds of debate; Step b5: Repeat steps b2-b4 until one of the preset debate termination conditions is met; the debate termination conditions are preset in the second prompt word and can be automatically identified by the mediation model.

[0044] Step b6: The mediation model integrates the structured memory and debate log to generate and output a unified image description text.

[0045] The unified image description text is the output of the mediation model after comprehensively synthesizing the debate process. It may still contain conflicts on key issues. The unified image description text will serve as semantic guidance for medical image analysis and will be used for subsequent disease classification through visual-language fusion.

[0046] Since the debate process generates visual descriptions, this naturally provides interpretability for the method of this invention, and the analysis and decision-making can be corroborated by these descriptions. For example... Figure 3 A unified image description text example is presented, along with an expert-generated radiology report as a reference, which provides a professional summary of key image appearances relevant to the disease. It can be seen that the multi-agent visual-language debate framework successfully identified relevant pathological features in the images, and the generated description is highly consistent with the reference report. Key disease-related content consistent between the MAVLD output and the report is highlighted (identical highlighted areas indicate consistent content).

[0047] By leveraging the relatively accurate unified image description text generated through multi-agent debate, the network model is able to better understand the images and extract disease-related image features more effectively.

[0048] Furthermore, this invention proposes a learnable visual-semantic fusion analysis model to align multi-level visual features with hierarchical semantic embeddings of a unified image description, thereby achieving multi-level, fine-grained capture of disease-related features in medical images and improving the interpretability and accuracy of the analysis results.

[0049] The visual-semantic fusion analysis model includes a text encoder, an image encoder, a multi-level fusion unit, and a classification head, wherein: A text encoder is used to encode the unified image description text to obtain multi-level text semantic features; An image encoder is used to encode the medical image to be analyzed and extract multi-level visual features; The multi-level fusion unit is used to perform cross-modal fusion of corresponding visual features and textual semantic features at multiple predetermined semantic levels to generate enhanced multimodal feature representations. A classification head is used to output the final disease analysis results based on the enhanced multimodal features.

[0050] Furthermore, prior to cross-modal fusion, for each predefined semantic level, the original visual feature map extracted from the image encoder is converted into a visual token consistent with the text embedding dimension of the corresponding level through linear projection and average spatial pooling.

[0051] Furthermore, the cross-modal fusion is achieved using a text-to-image cross-attention mechanism. For each semantic level, text embeddings are used as queries, and visual tokens are used as keys and values ​​to generate semantically guided enhanced visual features. The enhanced visual features are then integrated with the text embeddings to obtain a multimodal feature representation.

[0052] More specifically, the cross-modal fusion processing of the multi-level fusion unit at each predetermined semantic level includes: Step c1: Extract the original visual feature map from the image encoder. By using a learnable linear projection layer, the channel dimension is projected to be completely consistent with the text embedding dimension, thus obtaining the projected features. The linear projection layer is a 1×1 convolutional kernel or a fully connected layer, with a weight matrix of... ,in To represent the text embedding dimension, for example, take It equals 768. Indicates vision, Indicates projection. Indicates the first The original visual features of the layer This represents the visual feature dimension, i.e., the number of channels. Indicates the height of the visual feature map. This represents the width of the visual feature map.

[0053] Step c2, visual features after projection Applying global average pooling, visual grids are converted into visual tokens. More specifically, the visual spatial dimension Compressed to 1, a visual token is obtained. ; Here, visual token It is a compact representation of the global semantics of the image at this level, directly inherited from the original visual feature map output by the aforementioned image encoder. Each level independently generates a token, realizing hierarchical information decoupling and fusion. Indicates the first Projected feature map of the layer.

[0054] Step c3, embedding text As a query, with visual tokens As keys and values, cross-attention between image and text information is applied to generate semantically guided enhanced visual features. This cross-attention mechanism allows textual semantic information to guide the image encoder to focus on disease-related regions and features, suppressing interference from irrelevant background information, thereby improving the accuracy and relevance of image analysis.

[0055] Step c4 involves applying residual connections and layer normalization to integrate the enhanced visual features and text embeddings, obtaining the multimodal feature representation of this layer, and then passing it to subsequent layers of the text encoder. The formula includes: , in, Indicates semantic level, Indicates the first Level text embedding, Indicates the first Level visual tokens, Indicates the semantically guided first Enhanced visual features at the level of This represents multimodal feature representation. Representation layer normalization.

[0056] In step c5, the classification head inputs the [CLS] token from the last layer of the text encoder into a multilayer perceptron (MLP) to obtain the final analysis result of the medical image to be analyzed.

[0057] Furthermore, the final analysis results include the disease category corresponding to the medical image being analyzed.

[0058] Experimental tests and data To further illustrate the beneficial effects of the present invention, this embodiment provides a specific implementation scheme, test dataset, and test results for experimental verification of the present invention.

[0059] In the experimental verification phase of this invention, all patient data involved are publicly accessible datasets, and the images are preprocessed as follows: for 3D images, the axial slice with the largest segmentation area is obtained using standard segmentation results, and uniformly cropped to a size of 224x224; for 2D images, they are directly cropped to a size of 224x224 for use.

[0060] The framework designed in this invention and the comparative methods involved in the experiments were all written in Python, version 3.10. The experiments were deployed on three NVIDIA RTX 3090 GPUs, running Ubuntu 22.04. In Phase 1 and Phase 2, MedGemma (4B), Lingshu (7B), and Chiron-o1 (8B) were used as multimodal large-scale language model agents, and DeepSeek-V3.2 was selected as the mediator and prompt word generation model. These multimodal large-scale language models and large-scale language models were not retrained, and the number of debate rounds was... The threshold is set to 3. Stage 3 uses a 27-layer MedGemma visual tower and a 12-layer RoBERTa-base, with the low, medium, and high levels of image / text encoding selected as layers 7 / 2, 15 / 6, and 27 / 11, respectively. The multilayer perceptron head contains two layers with a hidden layer dimension of 3072, using the GELU activation function and layer normalization. It's important to note that the MedGemma image encoder is frozen; the entire model only needs to learn the weights of RoBERTa, cross-attention, and the multilayer perceptron. The AdamW optimizer is used to minimize the cross-entropy loss to learn the trainable model weights, with a learning rate set to 5×10⁻⁶. 5. The weight decay is 0.01, the number of training rounds is 40, and the batch size is 16.

[0061] This invention was tested on two publicly available datasets involving different tasks: brain tumor classification and chest disease classification. For the first task, the BraTS dataset was used, containing three cohorts: glioblastoma, brain metastases, and meningiomas. T1-weighted scan images were used because they provide high-resolution anatomical details and are the most commonly used modality in standard clinical protocols, ensuring the broad applicability of the method. For the second task, a random subset of the NIH-CXR dataset was used. This dataset contains X-ray images, focusing on five important diseases: infiltration, pleural effusion, atelectasis, nodules, and pneumothorax. Each dataset was divided into training, validation, and test sets. Detailed dataset partitioning for each disease type is shown in Table 3.

[0062] Table 3 summarizes the dataset composition, including disease types and data segmentation.

[0063] This invention's experimental tests compared the multi-agent visual-language debate framework with several commonly used methods, including image classification models and multimodal large-scale language models. Image classification models included state-of-the-art ResNet and ViT. Additionally, BiomedCLIP, pre-trained on biomedical image-text pairs, was selected; this model is capable of semantic medical image understanding and is widely used in medical image classification. Furthermore, this invention was compared with popular open-source multimodal large-scale language models, including the general-purpose Qwen3VL (8B) and biomedical-specific models such as MedGemma (4B), Lingshu (7B), Chiron-o1 (8B), and HuatuoGPT (7B). These multimodal large-scale language models generate disease classification results using prompts. All comparison models were fine-tuned using the same data partitioning as the multi-agent visual-language debate framework, employing its default hyperparameters and pre-trained weights. Except for BiomedCLIP, the image classification model underwent full parameter fine-tuning; due to the large size of the multimodal large-scale language model and the BiomedCLIP model, LoRA fine-tuning was used.

[0064] Classification performance was quantitatively evaluated using accuracy (Acc), area under the receiver operating characteristic (AUC), and F1 score (F1). It is important to note that the AUC of the multimodal large language model cannot be calculated because it only generates text indicating disease type. Table 4 presents the results for each task and the average performance. The multi-agent visual-language debate framework significantly outperformed all comparable methods on both tasks. On average, the multi-agent visual-language debate framework achieved approximately 9 percentage points higher accuracy and F1 score than the best competitor (MedGemma), and its AUC was more than 4 percentage points higher than ConvNeXt (the best-performing comparable method in terms of AUC).

[0065] Table 4: Performance comparison between the multi-agent visual-language debate framework and the contrastive model (best results are shown in bold, and second-best results are shown in underline).

[0066] Example 2 This embodiment provides an analysis system employing any of the methods described in Embodiment 1. The system includes: The data acquisition module is used to acquire the medical image to be analyzed, the first prompt word generation instruction, the target disease name, and the second prompt word; The first prompt word generation module is used to generate a first prompt word for the target disease based on the first prompt word generation instruction and the type of medical image to be analyzed. The multi-agent module is used to consult multiple independent agents with the medical image to be analyzed and the first prompt word as input, obtain multiple initial image description texts, and respond to the query instructions generated by the mediation model in the iterative debate process; The iterative debate and consensus generation module is used to coordinate the multiple agents through a mediation model based on the second prompt word, to conduct iterative debates starting from the multiple initial image description texts, resolve conflicts and reach consensus, and finally generate a unified image description text. The visual-language fusion analysis module is used to extract and fuse features from the unified image description text and the medical image to be analyzed, and output the disease analysis results of the medical image to be analyzed. The intelligent agent is a multimodal large-scale language model, and the mediation model is a large-scale language model.

[0067] Example 3 This embodiment uses a debate using three agents to illustrate the unified image description generation process. The models used by each agent and the mediation model are as follows: Agent A: MedGemma Agent B: Lingshu Agent C: Chiron-o1, Mediation model: DeepSeek V3.2.

[0068] The input consists of the medical image to be analyzed, the first prompt word generation command, the target disease name, and the second prompt word input. Driven by these inputs, the medical image analysis is automatically completed according to the three-stage workflow of the MAVLD framework. The key steps and process outputs are shown below for easy understanding: Step d1: Obtain the preset first prompt word generation instruction and determine the name of the target disease to be diagnosed, as shown in Table 5.

[0069] Table 5: Examples of First Prompt Word Generation Templates

[0070] Step d2: Input the preset first prompt word generation instruction and the target disease name into the prompt word generation model (LLM, such as DeepSeek) to generate the first prompt word for the target disease, as shown in Table 6.

[0071] Table 6: Examples of the first prompt words generated by the prompt word generation model

[0072] In step d3, the generated first prompt word and the original medical image to be analyzed are input into three multimodal large language model agents (doctors), respectively. The initial image description text output by the three agents is shown in Table 7.

[0073] Table 7: Examples of Agent Outputs

[0074] Step d4: Based on the second prompt word, the multiple agents are coordinated through a mediation model to conduct iterative debates starting from the multiple initial image description texts, resolve conflicts, reach consensus, and finally generate a unified image description text.

[0075] Table 8 shows examples of the second prompt words.

[0076] Table 8: Second cue words Example

[0077] The mediation model, based on the second cue word, begins with the initial image description output by the three agents in step d3 and initiates an iterative debate control process. During this process, relevant information is maintained through debate logs and structured memory. Table 9 shows an example of the mediation model's output recorded in the debate log after the second round, which includes the inference result, new locked facts (i.e., consensus facts), new conflicts (i.e., active conflicts awaiting verification), the target speaker (the target agent for the next round's query instruction), and the query instruction (the next round's instruction).

[0078] Table 9: Examples of Mediation Model Output

[0079] Step 5: Complete the iterative debate according to the preset number of debate rounds (e.g., three rounds), or meet other debate termination conditions. The mediation model summarizes the debate process, i.e., provides a unified image description, as shown in Table 10, which may still include unresolved conflicts.

[0080] Table 10: Examples of debate summaries output by the mediation model

[0081] In another implementation, the solution can be implemented as a device, which may include corresponding modules that perform one or more steps in the various embodiments described above. Therefore, each or more steps in the various embodiments can be performed by a corresponding module, and the electronic device may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0082] This device can be implemented using a bus architecture. A bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. The bus connects various circuits, including one or more processors, memory, and / or hardware modules. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuitry, external antennas, etc.

[0083] A bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but this does not imply that there is only one bus or one type of bus.

[0084] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this solution includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which the embodiments of this solution pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this solution can be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program can be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0085] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0086] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0087] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A medical image analysis method based on multi-agent visual-language debate, characterized in that, include: Step 1: Obtain the medical image to be analyzed, the first prompt word generation instruction, the target disease name, and the second prompt word; Step 2: Generate a first prompt word for the target disease based on the first prompt word generation instruction and the type of medical image to be analyzed; Step 3: Using the medical image to be analyzed and the first prompt word as input, consult multiple independent agents to obtain multiple initial image description texts; Step 4: Based on the second prompt word, the multiple agents are coordinated through the mediation model to conduct iterative debates starting from the multiple initial image description texts, resolve conflicts and reach consensus, and finally generate a unified image description text. Step 5: Perform feature extraction and feature fusion analysis on the unified image description text and the medical image to be analyzed, and output the disease analysis results of the medical image to be analyzed; The intelligent agent is a multimodal large-scale language model, and the mediation model is a large-scale language model.

2. The analytical method according to claim 1, characterized in that, The first prompt word generation instruction includes the agent role positioning, task objective, generation content specifications, output data format requirements, and target disease name placeholders, which are used to guide the large language model to generate the first prompt word for the medical image type and the target disease.

3. The analytical method according to claim 1, characterized in that, The second prompt includes the mediator model's role positioning, task objectives, permission boundaries, debate steps, debate termination conditions, and debate output data requirements. It is used to guide the mediator model to iteratively debate multiple initial image description texts and output a unified image description text.

4. The analytical method according to claim 3, characterized in that, The debate steps include: Step a1: Organize and merge the facts of consensus reached by all agents according to clinical themes; Step a2: Identify unresolved conflicts, name the conflict topics, and record the perspectives of each agent; Step a3: Prioritize conflicts and identify the conflicts that have the greatest impact on the final analysis results; Step a4: Strategic questioning and guidance to design the next query instruction for each key conflict and identify the target query object; Step a5: Decision-making and controlled convergence. When conflicts cannot be resolved, the majority opinion is adopted, and the uncertainty and reasons are recorded.

5. The analytical method according to claim 3, characterized in that, The debate output data is required to include a specified data format and data structure definition. The data structure definition includes the consensus facts generated in this round of debate, the conflicts to be resolved, the target intelligent agent for the next round of query, and the query instructions.

6. The analytical method according to claim 1, characterized in that, The control process for the iterative debate includes: Step b1, initialize the debate log and structured memory settings, including: ， ， in, , , This indicates the initial debate log. Represents the initial structured memory. This represents the initial image description set. Indicates the first cue word, Indicates the first An intelligent agent. The index number representing the intelligent agent. This represents the total number of intelligent agents. Indicates the first The initial image description text generated by the agent. This indicates the medical images to be analyzed. Step b2, for the first In each round of debate, the mediation model updates the consensus fact set and the active conflicts to be verified based on the debate logs from the previous round, structured memory, and preset second cue words, and generates query instructions for specific conflicts to be issued to the target agent, which can be expressed by the following formula: ， , in, Representing the mediation model, Indicates the second cue word, This refers to the log of the previous round of debate. This indicates an updated structured memory following the previous round of debate. This represents the structured memory updated after the current round of debate. Indicates the first The set of consensus facts generated by the rounds of debate Indicates the first The unverified active conflicts generated by the rounds of debate Indicates the first The query command generated by the round of debate, Indicates the current round of debate. It is a natural number greater than 0; Step b3: The agent re-analyzes the medical image to be analyzed according to the query instruction and generates a new image description, expressed by the formula: ， in, Indicates the first The agent in the th... The response of the wheel; Step b4, update the debate log for the current round, expressed by the formula: ， in, This indicates that all agents are in the 1st... A collection of image description texts obtained after rounds of debate; Step b5: Repeat steps b2-b4 until one of the preset debate termination conditions is met; Step b6: The mediation model integrates the debate log and structured memory to generate and output a unified image description text.

7. The analytical method according to claim 1, characterized in that, A visual-semantic fusion analysis model is used to extract and fuse features from the unified image description text and the medical image to be analyzed, outputting the final disease analysis results. The visual-semantic fusion analysis model includes a text encoder, an image encoder, a multi-level fusion unit, and a classification head, wherein: A text encoder is used to encode the unified image description text to obtain multi-level text semantic features; An image encoder is used to encode the medical image to be analyzed and extract multi-level visual features; The multi-level fusion unit is used to perform cross-modal fusion of corresponding visual features and textual semantic features at multiple predetermined semantic levels to generate enhanced multimodal feature representations. A classification head is used to output the final disease analysis results based on the enhanced multimodal features.

8. The analytical method according to claim 7, characterized in that, Before cross-modal fusion, for each predefined semantic level, the raw visual feature map extracted from the image encoder is converted into a visual token consistent with the text embedding dimension of the corresponding level through linear projection and average spatial pooling.

9. The analytical method according to claim 8, characterized in that, The cross-modal fusion is achieved using a text-to-image cross-attention mechanism. For each semantic level, text embedding is used as the query and visual tokens are used as the key and value to generate semantically guided enhanced visual features. The enhanced visual features are then integrated with the text embedding to obtain a multimodal feature representation.

10. An analysis system employing any one of the methods described in claims 1-8, characterized in that, include: The data acquisition module is used to acquire the medical image to be analyzed, the first prompt word generation instruction, the target disease name, and the second prompt word; The first prompt word generation module is used to generate a first prompt word for the target disease based on the first prompt word generation instruction and the type of medical image to be analyzed. The multi-agent module is used to consult multiple independent agents with the medical image to be analyzed and the first prompt word as input, obtain multiple initial image description texts, and respond to the query instructions generated by the mediation model in the iterative debate process; The iterative debate and consensus generation module is used to coordinate the multiple agents through a mediation model based on the second prompt word, to conduct iterative debates starting from the multiple initial image description texts, resolve conflicts and reach consensus, and finally generate a unified image description text. The visual-language fusion analysis module is used to extract and fuse features from the unified image description text and the medical image to be analyzed, and output the disease analysis results of the medical image to be analyzed. The intelligent agent is a multimodal large-scale language model, and the mediation model is a large-scale language model.

Images