Multi-agent inquiry method and system fusing fuzzy semantics and focus image features

Abstract

The invention discloses a multi-agent inquiry method and system fusing fuzzy semantics and focus image features. The method comprises the following steps: preprocessing a patient subjective description text and an original medical image to obtain a cleaned text and an enhanced image; analyzing the quantized text fuzzy semantics to generate a fuzzy semantic vector, guiding a visual model through the vector to extract focus features of an enhanced image, and generating a fuzzy visual prompt and attention thermodynamic diagram; then forming a consensus diagnosis result by multi-agent collaborative inquiry and debate reasoning based on the information; and finally, combining a diagnosis result, a thermodynamic diagram and a debate process to generate a structured report containing a diagnosis conclusion, visual evidence and a reasoning path. According to the method, accurate fusion of subjective fuzzy semantics and objective image features is realized, the medical illusion rate is reduced, the interactivity, interpretability and accuracy of diagnosis are improved, and the method is suitable for various intelligent medical auxiliary diagnosis scenes.

Description

Technical Field

[0001] This application belongs to the interdisciplinary field of artificial intelligence and medical information processing, specifically involving a multi-agent diagnostic method and system that integrates fuzzy semantics and lesion image features. Background Technology

[0002] With the rapid iteration of deep learning technology, the application of artificial intelligence (AI) in the healthcare field has moved from single-modal data processing to complex multimodal reasoning. However, in current clinical practice, existing medical AI products and methods still face severe technical bottlenecks, mainly reflected in the semantic gap between subjective descriptions and objective features, the illusory risk of monolithic models, and the uninterpretability of the diagnostic process.

[0003] (1) Limitations of existing single-modal and multi-modal diagnostic technologies: In traditional computer-aided diagnosis (CAD) systems, the technological approach is often fragmented into two isolated systems: image processing and natural language processing. Early CAD systems primarily relied on convolutional neural networks (CNNs) to segment or classify lesions in medical images (such as X-rays, CT scans, MRIs, and dermoscopy images). Although architectures like ResNet and DenseNet perform well on specific datasets, these systems are typically "black boxes," only outputting predicted probabilities, lacking semantic explanations of the decision-making process, and unable to integrate the patient's chief complaint for a comprehensive assessment.

[0004] Recently, large multimodal models (LMMs), such as GPT-4V, Gemini 2.5 Pro, and LLaVA-Med, have attempted to break down this barrier by aligning image encoders and text decoders through cross-modal connectors. However, existing research indicates that these general-purpose multimodal models suffer from a significant "subjective-objective semantic gap" in serious medical scenarios. The lack of quantification of ambiguous semantics: Patients' descriptions of symptoms are often unstructured, highly subjective, and full of ambiguity in natural language, such as vague descriptions like "a dull ache," "this redness seems to be spreading a bit," or "the lump feels a little bigger than yesterday." In linguistics, these descriptions contain a large number of ambiguous linguistic variables. Existing multimodal alignment techniques, such as CLIP and BLIP-2, are mainly based on contrastive learning of global features, tending to capture explicit objects in images but struggling to understand the mapping relationship between degree adverbs and subtle image features. When patients describe "blurred lesion edges," general large models often fail to accurately map this ambiguous concept to high-frequency signal attenuation areas at the pixel level of the image, causing the attention mechanism to fail to focus on clinically significant subtle lesions.

[0005] Lack of domain-specific visual localization capabilities: General-purpose LMMs perform well on natural scene images, but often exhibit "phantom localization" when processing high-resolution, low-contrast medical images. Studies indicate that when models are asked to point out "tiny nodules" in X-rays, due to a lack of deep understanding of medical anatomy and pathological features, they may incorrectly highlight blood vessel sections or artifacts. This visual localization bias directly undermines doctors' trust in AI-assisted diagnosis.

[0006] (2) Illusion and Reasoning Defects of Single-Agent Systems: Currently, most medical AI applications adopt a single-agent architecture, meaning that a single model completes the entire process from information intake to diagnosis generation. This architecture exposes fatal flaws when dealing with complex cases: Knowledge Consolidation and Factual Illusion: The knowledge of a single-unit LLM is limited by pre-training data and is easily influenced by the design of prompt words. When faced with rare cases, multiple complications, or incomplete information, the model tends to generate high-frequency words based on statistical probability rather than rigorous pathophysiological logical reasoning. This phenomenon is known as "medical illusion," manifesting as fabricating non-existent symptoms, citing incorrect medical guidelines, or generating false references. Research data shows that in complex clinical question-answering tasks, the error rate of single-unit models can be as high as 20%-30%, and they lack self-correction mechanisms. Once the initial reasoning path is flawed, subsequent diagnostic suggestions will produce cascading errors.

[0007] Lack of critical thinking and multi-perspective verification: In real clinical practice, multidisciplinary team (MDT) consultation is the standard procedure for solving complex cases, involving collaboration and verification among multiple roles such as radiologists, pathologists, and attending physicians. Single agents cannot simulate this complementary perspective among multiple roles. For example, radiologists focus on the morphological description of imaging features, while clinicians focus on the correlation between symptom evolution and past medical history. Single-agent models attempting to "wear many hats" often lead to neglecting certain aspects and failing to identify logical contradictions between imaging features and clinical symptoms.

[0008] (3) Lack of interactive diagnostic capabilities. Most existing auxiliary diagnostic systems are static, i.e., "input image + text = output diagnosis". However, real diagnosis and treatment is a dynamic, multi-round interactive process. Doctors need to ask patients about their medical history based on image features and revise their interpretation of the images based on the patients' answers.

[0009] Existing AI systems lack the ability to proactively plan diagnostic strategies. They typically passively accept all information input by the user, even if crucial information such as the duration of illness or family history is missing, and will still output a diagnosis rather than proactively inquire further. This passivity leads to a large number of misdiagnoses due to incomplete information. Although some studies have attempted to introduce reinforcement learning (RL) to optimize diagnostic strategies, they are often limited by predefined dialogue templates and lack the flexibility to handle complex interactions in natural language.

[0010] (4) Current Status and Limitations of Fuzzy Logic in Medical AI. Fuzzy logic, due to its ability to handle uncertainty and fuzziness, has long been used in expert systems. However, existing research on medical AI incorporating fuzzy logic shows a significant lag: The rigidity of the combination methods: Existing research mostly focuses on using fuzzy C-means algorithms for low-level image segmentation or using adaptive neurofuzzy inference systems to process structured tabular data. These methods are difficult to handle unstructured natural language text.

[0011] The gap in integration with large models: Currently, few studies directly map the membership function of fuzzy logic to the cue word engineering or attention weights of large language models. How to utilize fuzzy semantics to expand the hard reasoning boundaries of models and make them more consistent with human doctors' understanding of fuzzy semantics remains an underexplored area.

[0012] In summary, existing technologies lack a comprehensive diagnostic method that can effectively integrate fuzzy linguistic descriptions and lesion image features, and reduce misdiagnosis rates and improve diagnostic transparency through multi-role intelligent agent collaboration and verification. This application proposes an innovative solution to address the aforementioned technical pain points. Summary of the Invention

[0013] This application aims to address the following core technical problems existing in current medical auxiliary diagnostic technologies: Alignment Challenges Between Fuzzy Semantics and Visual Features: This application addresses the problem that vague language in patient complaints (such as "severe," "large area," and "vague") cannot accurately guide models to focus on specific lesion areas in medical images. Existing technologies struggle to map fuzzy metrics in natural language to attention weights in computer vision, leading to mismatches between text and images in diagnostic assessments. This application aims to establish a cross-modal alignment mechanism based on fuzzy sets.

[0014] Addressing the issues of illusion and unreliable reasoning in single-agent models: This application aims to resolve the problem that single LLMs are prone to factual errors (illusions) and lack self-correction capabilities when faced with complex cases. Existing technologies lack adversarial debate and consensus mechanisms similar to multi-agent medical teams (MDTs). This application introduces a criticism and reflection mechanism through a multi-agent collaborative architecture, significantly reducing the illusion rate.

[0015] Static diagnosis lacks interactivity and proactivity: This application addresses the problem that traditional CAD systems can only passively analyze images and cannot proactively initiate multiple rounds of targeted inquiries with patients to obtain key information for differential diagnosis, like a real doctor. It employs an information entropy-driven proactive inquiry strategy to simulate a real doctor-patient dialogue process.

[0016] The "black box" unexplainability of the diagnostic process: addressing the problem that deep learning models cannot explain "why it is judged as malignant" using logic that humans can understand.

[0017] This application aims to provide a transparent reasoning path across the entire chain through interpretability rules of fuzzy logic, visualized attention heatmaps, and dialogue logs of multi-agent systems.

[0018] To address the aforementioned technical problems, this application provides the following technical solutions.

[0019] Firstly, this application provides a multi-agent diagnostic method that integrates fuzzy semantics and lesion image features, comprising the following steps: S1. Preprocess the subjective description text input by the patient and the original medical images to obtain the cleaned subjective description text and the enhanced medical images. S2. Perform fuzzy semantic parsing and quantization on the cleaned subjective description text to generate fuzzy semantic vectors; S3. Use the fuzzy semantic vector to guide the visual model to extract lesion image features from the enhanced medical images, and generate fuzzy visual cues and attention heatmaps. S4. Based on the fuzzy semantic vector, fuzzy visual cues, and attention heatmap, a multi-agent collaborative diagnosis and debate reasoning is conducted to obtain a multi-agent consensus diagnosis result. S5. Based on the diagnostic results of the multi-agent consensus, the attention heatmap, and the multi-agent debate process, generate a structured diagnostic report containing diagnostic conclusions, visual evidence, and reasoning paths.

[0020] In some possible implementations, the preprocessing of S1 includes: Stop word removal and medical entity recognition are performed on the subjective description text to obtain the cleaned subjective description text; Adaptive histogram equalization and denoising are performed on the original medical images to obtain enhanced medical images.

[0021] In some possible implementations, the fuzzy semantic parsing and quantization of S2 includes the following sub-steps: S2.1 Extract fuzzy linguistic variables and corresponding modifiers from the cleaned subjective description text to form fuzzy linguistic variable-modifier binding pairs; S2.2 For each fuzzy language variable-modifier binding pair, retrieve the standard fuzzy language set corresponding to the fuzzy language variable, calculate the cosine similarity between the corresponding modifier and each word in the standard fuzzy language set, convert it into a normalized membership value, and take the word corresponding to the maximum membership value as the target word of the corresponding fuzzy language variable dimension. S2.3. Generate multi-dimensional fuzzy semantic vectors based on each target word and its corresponding membership value.

[0022] In some possible implementations, in S2.2, the standard fuzzy language set is a predefined set of professional terms for different fuzzy language variables in the medical field; the cosine similarity is converted into a normalized membership value through the Softmax function.

[0023] In some possible implementations, the lesion image feature extraction of S3 includes the following sub-steps: S3.1. The enhanced medical image is segmented into a sequence of image patches using a visual model based on the Transformer architecture. Multi-layer self-attention calculation and feature fusion are performed on the image patch sequence to obtain the fused image patch features. The fused image patch features are then integrated according to the spatial position of the image patches in the enhanced medical image to generate a multi-scale initial visual feature map. S3.2 Map the fuzzy semantic vector to a fuzzy weight matrix. The fuzzy weight matrix is ​​used to enhance or suppress different regions of the initial visual feature map according to the indication of the fuzzy semantic vector. Based on the fuzzy weight matrix, perform fuzzy weighted cross-attention calculation on the initial visual feature map to obtain the optimized visual features. S3.3 Perform structured analysis and visualization processing on the optimized visual features to generate fuzzy visual cues and generate an attention heatmap on the original medical images.

[0024] In some possible implementations, step S3.2 involves performing fuzzy weighted cross-attention calculation on the initial visual feature map to obtain optimized visual features, including: The query, key, and value vectors are obtained from the initial visual feature map through linear transformation. , , Then, the optimized visual features are calculated based on the following formula: ; in, For optimized visual features, The learnable adjustment coefficient, for Dimensions.

[0025] In some possible implementations, in step S4, the multi-agent system includes an attending physician agent, an imaging expert agent, a patient simulation agent, and a supervisory agent; collaborative consultation and debating reasoning through multiple agents includes the following sub-steps: S4.1 The imaging expert agent receives enhanced medical images, optimized visual features, and fuzzy visual cues, and outputs a structured imaging report. The attending physician agent combines the cleaned subjective description text, fuzzy semantic vectors, and imaging report, and retrieves relevant cases from the medical knowledge base through retrieval enhancement generation technology to generate a list of preliminary diagnostic hypotheses with confidence. S4.2 Calculate the information entropy of the preliminary diagnostic hypothesis list. If the entropy value is higher than the preset threshold, the attending physician agent generates proactive follow-up questions through decision tree logic to minimize the information entropy of the preliminary diagnostic hypothesis list. After interaction with the patient simulation agent, the diagnostic context is updated. S4.3 The attending physician agent proposes a final diagnostic conclusion based on the updated diagnostic context. The supervisory agent verifies the logical consistency between the diagnostic conclusion and the imaging report and patient description through a fuzzy reasoning system. If there is a contradiction, the diagnosis is rejected and the reasons are explained. The attending physician agent re-reasons or requests the imaging expert agent to re-examine the lesion area to supplement imaging evidence. The process iterates until a consensus is reached, and a multi-agent consensus diagnostic result is obtained.

[0026] In some possible implementations, the attending physician agent employs a thought chain prompt strategy to explicitly list evidence supporting and opposing the current diagnosis; the supervisory agent has a built-in medical domain-specific fuzzy rule library containing logical verification rules that correlate image features with clinical symptoms.

[0027] In some possible implementations, in S4.3, when the consensus of all agents exceeds a preset threshold or the maximum number of interaction rounds is reached, the iteration is stopped and the diagnostic result of the multi-agent consensus is determined.

[0028] In some possible implementations, in S5, the visual evidence is an original medical image superimposed with an attention heatmap, and the reasoning path is a multi-agent debate summary and differential diagnosis exclusion basis.

[0029] Secondly, this application provides a multi-agent diagnostic system that integrates fuzzy semantics and lesion image features, including: The data preprocessing module is used to preprocess the subjective description text input by the patient and the original medical images to obtain cleaned subjective description text and enhanced medical images. The fuzzy semantic parsing module is used to perform fuzzy semantic parsing and quantization on the cleaned subjective description text to generate fuzzy semantic vectors. The visual feature extraction module is used to guide the visual model to extract lesion image features from the enhanced medical images using fuzzy semantic vectors, and generate fuzzy visual cues and attention heatmaps. The multi-agent reasoning module is used to obtain a diagnostic result based on multi-agent consensus through collaborative questioning and debate reasoning based on fuzzy semantic vectors, fuzzy visual cues, and attention heatmaps. The report generation module is used to generate a structured diagnostic report containing diagnostic conclusions, visual evidence, and reasoning paths based on the diagnostic results of multi-agent consensus, attention heatmap, and multi-agent debate process.

[0030] Thirdly, this application provides an electronic device, including: a memory and a processor; The memory is used to store computer programs; The processor is used to invoke the computer program to execute the method described above.

[0031] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed on an electronic device, causes the electronic device to perform the method described above.

[0032] Fifthly, this application provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0033] The specific implementation methods of the second to fourth aspects of this application can refer to the implementation methods of the first aspect, and will not be elaborated here.

[0034] The multi-agent prompting diagnostic method and system proposed in this application, which integrates fuzzy semantics and lesion image features, has the following significant advantages over existing single-modal diagnostic systems or traditional static VQA medical models:

[0035] (1) Semantic-visual alignment capability

[0036] Pain points of existing technologies: Existing technologies (such as CLIP and BLIP) typically use rigid vector similarity matching, which is difficult to handle vague descriptions with degree adverbs such as "mild redness" or "dull pain," leading to the neglect of small or poorly defined lesions in the image. Traditional attention mechanisms are prone to losing weak semantic features.

[0037] Advantages of this application: By introducing fuzzy logic, this application quantifies fuzzy concepts in natural language into continuous membership values, and uses these as "soft attention weights" to directly apply to the feature extraction layer of the visual model. This fuzzy weighted cross-attention mechanism enables the model to "understand" the patient's subjective description and accurately locate the corresponding pathological areas in the image accordingly.

[0038] (2) Significantly reduces medical hallucinations and improves robustness

[0039] Existing technical challenges: When lacking information or operating in a knowledge blind spot, single-unit LLMs tend to fabricate seemingly professional but flawed diagnoses (illusions) and lack self-verification mechanisms. Single-unit models are also susceptible to bias due to prompts.

[0040] Advantages of this application: This application constructs a multi-agent debate architecture. The "supervisory agent" utilizes a rigid fuzzy rule base to perform logical consistency checks on the attending physician's "soft" reasoning. Research data shows that this multi-role mutual evaluation and consensus mechanism can reduce the rate of medical hallucinations by more than 30%, significantly improving the system's reliability when facing rare cases. Simultaneously, by introducing information entropy compression technology, redundant tokens in multi-turn dialogues are effectively reduced, improving reasoning efficiency.

[0041] (3) Simulate the interactive consultation of a real doctor

[0042] The pain points of existing technologies: Traditional CAD systems perform "one-off" static analysis, unable to eliminate interfering factors through follow-up questions like a human doctor. This passive reception of information leads to insufficient diagnostic evidence.

[0043] Advantages of this application: This application endows the intelligent agent with the ability to proactively ask questions. By calculating the information entropy of the current diagnostic hypothesis, the system can automatically determine when to obtain the maximum information increment by asking the patient follow-up questions (such as inquiring about medical history and family history). This "human-AI collaborative" interaction mode not only supplements the contextual information that images cannot provide, but also enhances the patient's sense of participation, resulting in a significant improvement in diagnostic accuracy after multiple rounds of dialogue, especially in complex cases where it demonstrates reasoning depth that surpasses that of single-model approaches.

[0044] (4) Enhance explainability and trustworthiness

[0045] Existing technological pain points: Deep learning models are often "black boxes," making it difficult for doctors and patients to understand why AI makes a diagnosis, which leads to difficulties in clinical application.

[0046] Advantages of this application: This application provides multidimensional interpretability.

[0047] Visual interpretability: The attention heatmap generated by fuzzy guidance visually shows the lesion areas that the model focuses on, proving that the model makes judgments based on correct visual evidence rather than background noise.

[0048] Logical interpretability: The dialogue logs and debate processes of the multi-agent system constitute a complete chain of reasoning. Users can see how different agents weigh evidence, eliminate dissent, and reach a consensus. This transparent reasoning path aligns with the clinical thinking logic of human doctors, greatly enhancing the system's transparency and clinical acceptability.

[0049] (5) Data privacy and deployment flexibility

[0050] Advantages of this application: The multi-agent architecture of this application supports federated learning extensions. The image expert agent can process sensitive image data locally, transmitting only fuzzy feature vectors and text reports to participate in central inference, thereby achieving collaborative diagnosis while protecting patient privacy. Furthermore, the fuzzy logic module has low computational resource requirements, allowing the system to be adapted to medical terminal devices with varying computing power.

[0051] This application creatively integrates fuzzy semantic computing with a multi-agent collaborative mechanism, systematically addressing the core pain points of traditional medical AI, such as the difficulty in fusing subjective and objective data, the black-box nature of the reasoning process, and the susceptibility to hallucinations. This technical solution not only theoretically achieves a deep unification of multimodal and fuzzy logic, but also demonstrates high precision, high reliability, and strong interpretability in clinical practice, representing the development direction of the next generation of intelligent assisted diagnostic systems. Attached Figure Description

[0052] Figure 1 This is a flowchart of a method according to one embodiment of this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be further described in detail below with reference to the embodiments and accompanying drawings.

[0054] It should be noted that the terms "comprising" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0055] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0056] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0057] This application provides a multi-agent diagnostic method and system that integrates fuzzy semantics and lesion image features. It belongs to the category of intelligent medical auxiliary diagnostic methods and systems based on the deep integration of Large Language Models (LLMs) and Computer Vision (CV). It utilizes fuzzy logic theory to process unstructured and highly subjective patient natural language descriptions, and through collaborative reasoning, adversarial debate, and interactive diagnostic mechanisms of a multi-agent system (MAS), it achieves accurate alignment, joint analysis, and interpretable decision-making between objective features of lesion images and patient subjective semantics. This application can be widely applied to scenarios such as electronic health record (EHR) analysis, intelligent triage platforms, remote medical image diagnosis, and dermatology and radiology decision support systems (CDSS).

[0058] Example 1: This embodiment provides a multi-agent prompting method for medical consultation that integrates fuzzy semantics and lesion image recognition. Fuzzy logic is used to quantize unstructured natural language into a computable membership vector, and this vector is used as a soft prompt, or soft prompt word, which is injected into the interaction process of the multi-agent and the attention mechanism of the visual model, thereby achieving a deep fusion of subjective perception and objective features.

[0059] like Figure 1 As shown, the method flow of this embodiment may include the following steps: S1. Preprocess the subjective description text input by the patient and the original medical images to obtain the cleaned subjective description text and the enhanced medical images.

[0060] The input data for this step consists of patient multimodal data, including subjective descriptive text (patient complaints) such as “persistent dull pain in the left lower abdomen, accompanied by mild nausea” and raw medical images such as CT scans, dermoscopy images, or X-rays.

[0061] In some embodiments, preprocessing, or data cleaning, includes: For subjective descriptive text: stop words are removed, Named Entity Recognition (NER) is performed, symptom entities and their modifiers are labeled, resulting in the cleaned subjective descriptive text. .

[0062] For the original medical images: Adaptive histogram equalization is applied to enhance contrast, and denoising is performed to reduce the interference of low-quality images on subsequent feature extraction, resulting in enhanced medical images. .

[0063] S2. Perform fuzzy semantic analysis and quantization on the cleaned subjective description text to generate fuzzy semantic vectors.

[0064] This step aims to transform ambiguous natural language into mathematical vectors that can be processed by computers. In some embodiments, it specifically includes: S2.1 Extract fuzzy linguistic variables and corresponding modifiers from the cleaned subjective description text to form fuzzy linguistic variable-modifier binding pairs.

[0065] We used a specially fine-tuned medical big language model (LLM) to analyze the cleaned subjective descriptive text, extract key fuzzy linguistic variables (used to describe symptom characteristics) and their corresponding modifiers, forming fuzzy linguistic variable-modifier binding pairs.

[0066] For example, extracting fuzzy linguistic variables from the description "the pain is quite severe". (Severity) is assigned the modifier "relatively severe"; fuzzy linguistic variables are extracted from "the edges of the erythema are unclear". (Border Clarity) is assigned the modifier "unclear", thus forming binding pairs, such as (Severity, relatively severe) and (Border_Clarity, unclear).

[0067] S2.2 For each fuzzy language variable-modifier binding pair, retrieve the standard fuzzy language set corresponding to the fuzzy language variable, calculate the cosine similarity between the corresponding modifier and each word in the standard fuzzy language set, convert it into a normalized membership value, and take the word corresponding to the maximum membership value as the target word of the corresponding fuzzy language variable dimension.

[0068] For each extracted binding pair containing fuzzy language variables, the corresponding standard fuzzy language set is retrieved (e.g., the set S={Mild,Moderate,Severe,Critical} corresponds to Severity).

[0069] In some embodiments, a predefined set of professional terms can be used as a standard fuzzy language set for different fuzzy language variables in the medical field. For example, for Severity, a standard fuzzy concept set defining the severity of symptoms can be used. This is used to quantify a patient's fuzzy description of the severity of symptoms, where Mild, Moderate, Severe, and Critical represent mild, moderate, severe, and critical symptoms, respectively; and SymptomTimeSpan, a fuzzy linguistic variable defining the symptom time span, defines the standard set of fuzzy concepts for defining the symptom time span. It is used to quantify patients' vague descriptions of symptom duration and onset speed, where Acute, Subacute, and Chronic represent acute, subacute, and chronic symptoms, respectively.

[0070] Membership degree calculation: Calculate the modifiers in the binding pair The text embedding vector of "(e.g., vaguely)" and the text embedding vectors of each word in the corresponding standard fuzzy set (e.g., Text embedding vector and Text embedding vector The cosine similarity of the two pairs is calculated using the Softmax function. Converted to normalized membership values For example, modifiers correspond Membership value Modifiers correspond Membership value The words corresponding to the maximum membership value are used as the matching words (target words) for the corresponding fuzzy linguistic variable dimensions.

[0071] This step innovatively combines the embedding space distance of LLM to dynamically generate membership functions (MFs), rather than using rigid fixed rules.

[0072] Fuzzy semantic vector generation: Based on the target vocabulary and confidence data of each fuzzy linguistic variable dimension, a multi-dimensional fuzzy semantic vector is generated. It is used to encode the degree, extent, uncertainty, and urgency of symptoms.

[0073] S3. Using the fuzzy semantic vector, guide the visual model to extract lesion image features from the enhanced medical images, and generate fuzzy visual cues and attention heatmaps.

[0074] This step addresses the "image-text alignment" problem by using the fuzzy semantic vectors generated by S2 to guide the visual model on "where to look." In some embodiments, this step may include: S3.1. Using a Transformer-based visual model, such as the BioMedGPT visual encoder, which is specifically optimized for the biomedical imaging field, the enhanced medical images are processed. The image is segmented into a sequence of image patches. Multi-layer self-attention computation and feature fusion are performed on the image patch sequence to obtain fused image patch features. These fused image patch features are then arranged according to the image patches... Spatial location integration within the model generates multi-scale initial visual feature maps. .

[0075] S3.2, fuzzy semantic vector Mapped to fuzzy weight matrix The fuzzy weight matrix is ​​used to enhance or suppress different regions of the initial visual feature map according to the indication of the fuzzy semantic vector; based on the fuzzy weight matrix For the initial visual feature map Fuzzy weighted cross-attention calculation is performed to obtain the optimized visual features.

[0076] In some embodiments, at the decoder layer or fusion layer of the Transformer, the initial visual feature map is used... This yields three learnable linear transformation matrices. , , Get the query vector Key vector Value vector .Will Mapped to a fuzzy weight matrix via a multilayer perceptron (MLP). Based on Q, K, V and Attention calculations are performed to achieve fuzzy weighted optimization of the initial visual features.

[0077] The traditional attention calculation formula is This application introduces fuzzy semantic vectors. As a regulatory factor The fuzzy weight matrix is ​​obtained through multilayer perceptron (MLP) mapping. ,pass The traditional attention calculation formula is modified, and the modified attention formula is as follows: ; in, For optimized visual features, The learnable adjustment coefficient, for The dimension. If If the boundary of the indicator is ambiguous, then It suppresses the weight of high-frequency edge features and enhances the weight of texture transition areas; if "localized severe pain" is indicated, the attention mechanism will force focus on the anatomical area associated with the distribution of pain nerves.

[0078] S3.3 Perform structured analysis and visualization processing on the optimized visual features to generate fuzzy visual cues. An attention heatmap is generated on the original medical images to visually display the lesion areas (regions of interest) that the model is focusing on.

[0079] S4. Based on the fuzzy semantic vector, fuzzy visual cues, and attention heatmap, a multi-agent collaborative diagnosis and debate reasoning is conducted to obtain a multi-agent consensus diagnosis result.

[0080] This application constructs a multi-agent system based on a multimodal large language model to simulate a multidisciplinary consultation scenario in a hospital.

[0081] In some embodiments, the following agents are initialized based on a preset structured prompt template: Doctor Agent: Responsibilities: To coordinate all information, initiate consultations, propose preliminary diagnostic hypotheses, and formulate treatment plans.

[0082] Configuration: Loads the Medical Guidelines Knowledge Base (Retrieval-augmented Generation, RAG), enabling logical reasoning and decision-making capabilities.

[0083] Cue word strategy: The Chain-of-Thought (CoT) cue word strategy is adopted, which requires the model to explicitly list the evidence supporting and opposing the current diagnosis.

[0084] Radiologist Agent: Responsibilities: Focus on image interpretation. Receive enhanced medical images. Optimized visual features and blurred visual cues .

[0085] Capabilities: Utilizing visual question answering (VQA) capabilities, image features can be transformed into professional radiological terms, such as "ground-glass opacity," "calcifications," and "marginal spiculations."

[0086] Output: Generates a structured radiographic report and provides probabilistic annotations for uncertain features.

[0087] Patient Agent: Responsibilities: To act as a data generator during the simulation training phase, or as a mediator for user interaction in practical applications.

[0088] Function: It is responsible for "translating" doctors' professional questions into easy-to-understand language and presenting them to real patients, and for standardizing patients' oral responses and feeding them back to the system.

[0089] Supervisor Agent: Responsibilities: Act as a "referee," responsible for monitoring dialogue logic, checking for LLM hallucinations, and assessing the consistency between physician diagnoses and imaging features.

[0090] Core algorithm: Utilizes a fuzzy inference system (FIS) as a logic checker.

[0091] Rule base: Built-in medical-specific fuzzy rule base, which contains logical verification rules that associate image features with clinical symptoms, such as "IF image shows solidification AND patient has no fever THEN bacterial pneumonia probability LOW" (meaning "if the image shows solidification and the patient has no fever, then the probability of bacterial pneumonia is low").

[0092] This step is the core dynamic interaction process of this application, and may include the following sub-processes: S4.1 Independent Analysis and Hypothesis Generation: The imaging expert agent, based on enhanced medical images Optimized visual features and blurred visual cues are used to output a structured radiological report.

[0093] The attending physician's intelligent agent combined with the cleaned subjective description text Fuzzy semantic vectors Based on imaging reports, relevant cases are retrieved from a medical knowledge base using Retrieval Enhancement Generation (RAG) technology to generate a preliminary list of diagnostic hypotheses with confidence levels. ,For example Melanoma (60%) Seborrheic keratosis (30%).

[0094] S4.2, Proactive questioning based on information entropy: This step addresses the initial assumptions from the first round by quantifying diagnostic uncertainty through information entropy. It proactively seeks key information from patients, resolving the pain points of traditional medical AI's passive reception of information and misdiagnosis due to incomplete information. The core operation is as follows: Uncertainty assessment: Calculate the current preliminary diagnostic hypothesis list The information entropy (the higher the entropy value, the more dispersed the confidence of each hypothesis, and the less clear the diagnosis). If the entropy value is higher than the preset threshold, it means that the diagnosis is still unclear, triggering the active questioning process.

[0095] Proactive follow-up question generation: The attending physician agent uses decision tree logic to generate proactive follow-up questions that minimize information entropy. The questions all point to key information for differential diagnosis, such as the duration of illness, family history, and whether the lesion has ulcerated / bleeded. For example, "Has the lesion ulcerated or bled in the past three months?"

[0096] Doctor-patient interaction and information updates: Proactively asking follow-up questions Send the information to the patient via the user interface, and update the diagnostic context after receiving the patient's feedback.

[0097] S4.3 Multi-agent debate and consensus reaching: Debate Trigger: The attending physician agent proposes a final diagnostic conclusion based on the updated diagnostic context (determining the core diagnosis from hypothesis list H), triggering the verification process of the critic / supervisor agent.

[0098] Logical verification: The critic agent compares the logical consistency of the diagnostic conclusion, imaging evidence (S3), and patient description (feedback from S1+S4.2). If a contradiction is found, the diagnosis is rejected and the reasons are explained. For example, if the doctor diagnoses "benign," but the imaging expert points out that "the boundary is extremely irregular," the supervisory agent outputs: "Diagnosis rejected. Reason: The imaging feature 'irregular boundary' conflicts with the membership degree of the benign diagnosis."

[0099] Adversarial correction: The attending physician agent receives a verification challenge, re-integrates complete evidence for reasoning, or requests the imaging expert agent to re-examine specific areas of the lesion based on the attention heatmap to supplement imaging evidence.

[0100] Consensus Mechanism: Iteratively perform the above logical verification-adversarial correction process until one of the following two conditions is met: ① The consensus of the conclusions of all agents exceeds a preset threshold; ② The maximum number of interaction rounds is reached; Finally, the diagnostic result verified by multiple agents is output, which is the diagnostic result of multi-agent consensus.

[0101] S5. Based on the diagnostic results of the multi-agent consensus, the attention heatmap, and the multi-agent debate process, generate a structured diagnostic report containing diagnostic conclusions, visual evidence, and reasoning paths.

[0102] This step generates a structured report and provides multi-dimensional interpretation, outputting a comprehensive report containing the following: Diagnostic conclusion: Disease prediction and confidence interval based on multi-agent consensus.

[0103] Visual evidence: An attention heatmap is overlaid on the original image, highlighting key pixel areas that support the diagnosis.

[0104] Reasoning path visualization: Displays a summary of the debate among the multi-agents, explaining which differential diagnoses were ruled out and the basis for the exclusion, such as: "Basal cell carcinoma was ruled out because the patient described no ulcers and the imaging showed an intact pigment network structure."

[0105] It should be understood that the numbers S1 to S5 above are only used to distinguish and facilitate the expression of different steps, and do not necessarily constitute a restriction on the execution order between the steps.

[0106] To more clearly illustrate the technical solutions of the above embodiments, the implementation and results of some steps are explained below in a simulated application scenario: auxiliary diagnosis of skin lesions.

[0107] Simulated application scenario background: A virtual patient uploaded a picture of irregular dark brown spots on their arm via a mobile app, along with the text description: "These dark spots have recently gotten bigger, and the color varies in shade. Sometimes they feel a little itchy, like being pricked by needles."

[0108] (1) Specific implementation of Fuzzy Semantic Mapping (FSMM): Input text analysis: Identify key entities and modifiers: Entity 1: "Black Spot" → Object (Lesion); Feature 1: "Getting bigger" → Evolution, modifier: "recently"; Feature 2: "Color" → Color, modifier: "various shades"; Feature 3: "Itching" → sensation, modifiers: "a little", "like needle pricks".

[0109] Fuzzy computation: Evolution: Mapping "recently increased" to the fuzzy linguistic variable evolution rate. The corresponding standard fuzzy language set is ,in These represent rapid evolution and stable invariance, respectively. Based on semantic distance calculation, the membership degrees for rapid evolution and stable invariance are obtained as follows: , .

[0110] Color: Maps "different shades" to the fuzzy linguistic variable color uniformity. The corresponding standard fuzzy language set is ,in Let represent uneven color and uniform color, respectively. Based on semantic distance calculation, the membership degrees for uneven color and uniform color are obtained as follows: , 1.

[0111] Sensation: Mapping "slight itching, tingling" to fuzzy linguistic variables related to somatosensory symptoms. The corresponding standard fuzzy language set is ,in These represent pain / sting and itching, respectively. Based on semantic distance calculation, the membership degrees of pain / sting and itching are obtained as follows: .

[0112] Vector output: Generate fuzzy feature vectors: .

[0113] (2) Specific implementation of visual feature extraction guided by fuzzy logic: Visual model: The BioMedGPT visual encoder, finely tuned on the ISIC dermatology dataset, is used.

[0114] Attention injection: The model receives the original image and .

[0115] because middle (Uneven color height) The model's cross-attention module automatically enhances the weights of the color channels.

[0116] because This suggests the instability of the boundary. The model uses a fuzzy edge detection operator to enhance attention to the jagged structure of the lesion edge and suppress the texture noise of the surrounding normal skin.

[0117] Results: An enhanced feature map was generated in which the blue-white curtain region inside the spot and the pseudopodia structure at the edge were significantly highlighted, which are key features of malignant melanoma.

[0118] (3) Implementation of the multi-agent collaborative interaction process: The following is the actual internal operating trajectory of the system: Initialization phase: The attending physician's AI agent is loaded with dermatology treatment guidelines.

[0119] Supervise the loading of a fuzzy inference system by an intelligent agent.

[0120] Round 1: The imaging expert AI agent reported: "The lesion is approximately 7mm in diameter, with asymmetrical borders, exhibiting multiple colors (black, brown, and blue), and irregular stripes visible at the edges. It corresponds to a relatively high ABCD score."

[0121] The attending physician's AI agent states: "Based on the patient's description of 'recent enlargement' (rapid evolution) and the 'multicolored, asymmetrical' features in the imaging, the preliminary diagnosis is: Malignant Melanoma, with a confidence level of 85%. Differential diagnosis: Dysplastic nevus."

[0122] Supervisory agent intervention: Supervisory agent: "Logic check: The doctor's diagnosis of melanoma is well-founded. However, it should be noted that the patient mentioned a 'pinching sensation.' This may indicate nerve involvement or simple inflammation. To rule out inflammatory skin conditions (such as pyogenic granuloma), it is necessary to confirm whether there is a history of trauma or bleeding."

[0123] Strategy (Action): The supervising agent instructs the doctor to initiate follow-up questions.

[0124] Interactive follow-up questions: The system asks the patient: "Has this dark spot been injured recently? Is there any bleeding or ulceration?"

[0125] The user replied: "I haven't been injured, but a few days ago my clothes rubbed together and there was a little bit of bleeding."

[0126] Round 2: The attending physician's AI agent stated: "The patient has been confirmed to have spontaneous bleeding (non-traumatic), which is one of the warning signs of advanced melanoma. Simple moles have been ruled out. The confidence level for melanoma has been increased to 95%."

[0127] The imaging expert agent stated: "Re-examine the bleeding area. The highlighting in the central area of ​​the attention heatmap confirms signs of ulceration."

[0128] Consensus reached: Supervisory agent: "Validation passed. The diagnosis is logically consistent with all evidence (images + descriptions + follow-up questions) and there are no signs of hallucinations."

[0129] (4) Result output

[0130] The system ultimately displays the following to the user: Conclusion: Highly suspected malignant melanoma.

[0131] Recommendation: Please visit a dermatologist within one week for a dermoscopy and excision biopsy.

[0132] Explanation: The AI ​​system detected a lesion that was rapidly increasing in size (evolving) within a short period, and the image showed dangerous color inconsistencies and boundary features. Signs of bleeding further increased the risk.

[0133] Example 2: This embodiment provides a multi-agent diagnostic system that integrates fuzzy semantics and lesion image features, including: The data preprocessing module is used to preprocess the subjective description text input by the patient and the original medical images to obtain cleaned subjective description text and enhanced medical images. The fuzzy semantic parsing module, also known as the fuzzy semantic mapping module (FSMM), is used to perform fuzzy semantic parsing and quantification on the cleaned subjective descriptive text, generating fuzzy semantic vectors. This module is responsible for converting the patient's natural language description into fuzzy semantic vectors, resolving the ambiguity of language.

[0134] The visual feature extraction module, also known as the Fuzzy-Guided Visual Attention Module (FG-VAM) or the Fuzzy-Guided Cross-Modal Attention Encoder, is used to guide a visual model to extract lesion image features from enhanced medical images using fuzzy semantic vectors, generating fuzzy visual cues and attention heatmaps. This module utilizes the attention map of a fuzzy semantic vector-weighted visual model (such as Vision Transformer, ViT) to address the misalignment between semantic and visual features.

[0135] The multi-agent reasoning module is used to achieve a consensus-based diagnostic result through collaborative questioning and debating reasoning among multiple agents, based on fuzzy semantic vectors, fuzzy visual cues, and attention heatmaps. This module is implemented using a Multi-Agent Collaborative Inquiry Network (MACIN) that includes agents such as the attending physician, imaging expert, and supervisory agents. It addresses the illusion and logical flaws inherent in single-agent reasoning.

[0136] The report generation module is used to generate a structured diagnostic report containing diagnostic conclusions, visual evidence, and reasoning paths based on the diagnostic results of multi-agent consensus, attention heatmap, and multi-agent debate process.

[0137] Example 3: This embodiment provides an electronic device, including: a memory and a processor; The memory is used to store computer programs; The processor is configured to invoke the computer program to execute the method as described in Embodiment 1.

[0138] Example 4: This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is run on an electronic device, it causes the electronic device to perform the method described in Embodiment 1.

[0139] Example 5: This embodiment provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described in Embodiment 1.

[0140] The specific implementation of the system, electronic device, computer-readable storage medium, and computer program product provided in this application can be referred to the specific embodiments of the above methods, and will not be repeated here.

[0141] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0142] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-agent consultation method fusing fuzzy semantics and lesion image features, characterized in that, Includes the following steps: S1. Preprocess the subjective description text input by the patient and the original medical images to obtain the cleaned subjective description text and the enhanced medical images. S2. Perform fuzzy semantic parsing and quantization on the cleaned subjective description text to generate fuzzy semantic vectors; S3. Use the fuzzy semantic vector to guide the visual model to extract lesion image features from the enhanced medical images, and generate fuzzy visual cues and attention heatmaps. S4. Based on the fuzzy semantic vector, fuzzy visual cues, and attention heatmap, a multi-agent collaborative diagnosis and debate reasoning is conducted to obtain a multi-agent consensus diagnosis result. S5. Based on the diagnostic results of the multi-agent consensus, the attention heatmap, and the multi-agent debate process, generate a structured diagnostic report containing diagnostic conclusions, visual evidence, and reasoning paths.

2. The multi-agent consultation method of fusing fuzzy semantics and lesion image features according to claim 1, characterized in that, The preprocessing of S1 includes: Stop word removal and medical entity recognition are performed on the subjective description text to obtain the cleaned subjective description text; Adaptive histogram equalization and denoising are performed on the original medical images to obtain enhanced medical images.

3. The multi-agent diagnostic method for fusing fuzzy semantics and lesion image features according to claim 1, characterized in that, The fuzzy semantic parsing and quantization of S2 includes the following sub-steps: S2.1 Extract fuzzy linguistic variables and corresponding modifiers from the cleaned subjective description text to form fuzzy linguistic variable-modifier binding pairs; S2.2 For each fuzzy language variable-modifier binding pair, retrieve the standard fuzzy language set corresponding to the fuzzy language variable, calculate the cosine similarity between the corresponding modifier and each word in the standard fuzzy language set, convert it into a normalized membership value, and take the word corresponding to the maximum membership value as the target word of the corresponding fuzzy language variable dimension. S2.

3. Generate multi-dimensional fuzzy semantic vectors based on each target word and its corresponding membership value.

4. The multi-agent diagnostic method for fusing fuzzy semantics and lesion image features according to claim 3, characterized in that, In step S2.2, the standard fuzzy language set is a predefined set of professional terms for different fuzzy language variables in the medical field; the cosine similarity is converted into a normalized membership value through the Softmax function.

5. The multi-agent diagnostic method for fusing fuzzy semantics and lesion image features according to claim 1, characterized in that, The lesion image feature extraction in S3 includes the following sub-steps: S3.

1. The enhanced medical image is segmented into a sequence of image patches using a visual model based on the Transformer architecture. Multi-layer self-attention calculation and feature fusion are performed on the image patch sequence to obtain the fused image patch features. The fused image patch features are then integrated according to the spatial position of the image patches in the enhanced medical image to generate a multi-scale initial visual feature map. S3.2 Map the fuzzy semantic vector to a fuzzy weight matrix. The fuzzy weight matrix is ​​used to enhance or suppress different regions of the initial visual feature map according to the indication of the fuzzy semantic vector. Based on the fuzzy weight matrix, perform fuzzy weighted cross-attention calculation on the initial visual feature map to obtain the optimized visual features. S3.3 Perform structured analysis and visualization processing on the optimized visual features to generate fuzzy visual cues and generate an attention heatmap on the original medical images.

6. The multi-agent diagnostic method for fusing fuzzy semantics and lesion image features according to claim 5, characterized in that, In step S3.2, fuzzy weighted cross-attention calculation is performed on the initial visual feature map to obtain optimized visual features, including: The query, key, and value vectors are obtained from the initial visual feature map through linear transformation. , , Then, the optimized visual features are calculated based on the following formula: ； in, For optimized visual features, The learnable adjustment coefficient, for Dimensions.

7. The multi-agent diagnostic method for fusing fuzzy semantics and lesion image features according to claim 5, characterized in that, In step S4, the multi-agent system includes an attending physician agent, an imaging expert agent, a patient simulation agent, and a supervisory agent. Collaborative consultation and debating reasoning through multiple agents includes the following sub-steps: S4.1 The imaging expert agent receives enhanced medical images, optimized visual features, and fuzzy visual cues, and outputs a structured imaging report. The attending physician agent combines the cleaned subjective description text, fuzzy semantic vectors, and imaging report, and retrieves relevant cases from the medical knowledge base through retrieval enhancement generation technology to generate a list of preliminary diagnostic hypotheses with confidence. S4.2 Calculate the information entropy of the preliminary diagnostic hypothesis list. If the entropy value is higher than the preset threshold, the attending physician agent generates proactive follow-up questions through decision tree logic to minimize the information entropy of the preliminary diagnostic hypothesis list. After interaction with the patient simulation agent, the diagnostic context is updated. S4.3 The attending physician agent proposes a final diagnostic conclusion based on the updated diagnostic context. The supervisory agent verifies the logical consistency between the diagnostic conclusion and the imaging report and patient description through a fuzzy reasoning system. If there is a contradiction, the diagnosis is rejected and the reasons are explained. The attending physician agent re-reasons or requests the imaging expert agent to re-examine the lesion area to supplement imaging evidence. The process iterates until a consensus is reached, and a multi-agent consensus diagnostic result is obtained.

8. The multi-agent diagnostic method for fusing fuzzy semantics and lesion image features according to claim 7, characterized in that, The attending physician agent employs a thought chain prompt strategy to explicitly list evidence supporting and opposing the current diagnosis; the supervisory agent has a built-in medical-specific fuzzy rule library, which contains logical verification rules for the association between image features and clinical symptoms.

9. The multi-agent diagnostic method for fusing fuzzy semantics and lesion image features according to claim 7, characterized in that, In S4.3, when the consensus of all agents exceeds a preset threshold or the maximum number of interaction rounds is reached, the iteration stops and the diagnostic result of the multi-agent consensus is determined.

10. A multi-agent diagnostic system integrating fuzzy semantics and lesion image features, characterized in that, include: The data preprocessing module is used to preprocess the subjective description text input by the patient and the original medical images to obtain cleaned subjective description text and enhanced medical images. The fuzzy semantic parsing module is used to perform fuzzy semantic parsing and quantization on the cleaned subjective description text to generate fuzzy semantic vectors. The visual feature extraction module is used to guide the visual model to extract lesion image features from the enhanced medical images using fuzzy semantic vectors, and generate fuzzy visual cues and attention heatmaps. The multi-agent reasoning module is used to obtain a diagnostic result based on multi-agent consensus through collaborative questioning and debate reasoning based on fuzzy semantic vectors, fuzzy visual cues, and attention heatmaps. The report generation module is used to generate a structured diagnostic report containing diagnostic conclusions, visual evidence, and reasoning paths based on the diagnostic results of multi-agent consensus, attention heatmap, and multi-agent debate process.

Images