Medical image collaborative analysis method and device based on parallel truncated fusion and morphological perception visual prompt

Abstract

The application provides a medical image collaborative analysis method and device based on parallel truncated fusion and morphology perception visual prompt, and relates to the technical field of medical image analysis. The method comprises the following steps: performing high-precision lesion segmentation on a medical image by using a parallel truncated fusion network. The network injects the deep semantics of a frozen large language model into a visual backbone in a residual manner through a non-destructive parallel double-flow design, enhances semantic understanding while retaining texture details; generating a morphology perception visual prompt based on the segmentation result, guiding a downstream visual language model to focus on the lesion area through a non-occlusion safety bounding box and a soft thinking chain, and completely retaining the environment features which are crucial for diagnosis. The application effectively improves the medical image segmentation performance and the accuracy of the medical VQA task.

Description

Technical Field

[0001] This application relates to the field of medical image analysis technology, and in particular to a method and apparatus for collaborative analysis of medical images based on parallel truncated fusion and morphological perception visual cues. Background Technology

[0002] In recent years, with the explosive growth of large-scale visual language models (LVLMs) such as GPT-4V and Qwen-VL, medical artificial intelligence is undergoing a significant transformation from single-task models to general-purpose clinical tools. These large models, leveraging massive amounts of pre-trained knowledge, have demonstrated powerful medical question-answering and logical reasoning capabilities. However, when directly applied to clinical imaging diagnostic scenarios with extremely high accuracy requirements, these general-purpose large models often face several challenges, including a lack of perceptual accuracy and computational efficiency bottlenecks. This challenge often manifests as attention drift; when faced with complex medical images, such as dermoscopy with hair interference or low-contrast ultrasound, the model often struggles to focus on minute lesions, leading to illusory diagnoses such as failing to notice or misidentifying lesions. Furthermore, running models with billions of parameters for pixel-level medical diagnosis incurs enormous memory overhead and latency, making it unbearable for clinical applications.

[0003] To address these issues, existing technologies have explored numerous pathways to decouple perception and cognition. Traditional convolutional neural networks (CNNs) offer fast inference and strong texture features but weak semantic features. Recent attempts to sequentially embed LLM layers into segmentation networks, while introducing semantic features from large language models, suffer from resolution mismatch, leading to oversmoothing of high-frequency features and obscuring key pathological features at lesion edges. More importantly, in building the bridge between perception and cognition, existing methods prioritizing localization before diagnosis often remain at a rudimentary validation stage. For example, some studies use localization information only as an evaluation metric or employ simple masking strategies as visual cues. While this occlusion-based guidance corrects attention, it severs the transmission of crucial features such as lesion edge infiltration and lobulation, resulting in the loss of evidence for benign / malignant differentiation. Furthermore, there is currently no integrated diagnostic framework that combines localization and diagnosis; previous localization information often relies on manual annotation, which is inefficient. Summary of the Invention

[0004] This application provides a medical image collaborative analysis method and device based on parallel truncated fusion and morphologically-aware visual cues. It designs a refined morphologically-aware collaborative diagnostic framework, aiming to elevate the localization-then-diagnosis paradigm to a highly integrated and practical level through a dual mechanism of non-destructive semantic injection and non-occluded visual guidance. First, for front-end perception, a parallel truncated fusion network (PTF-Net) is designed as an accurate and efficient observer for the system. Unlike previous destructive serial reconstructions, this application treats the frozen LLM as a plug-and-play external knowledge base, using a truncated extraction strategy to obtain only deep semantics, and explicitly injects it into the lightweight CNN backbone through a parallel trainable adapter and an innovative feature fusion module. This parallel fusion design completely avoids the problem of over-smoothing features while endowing the visual model with powerful semantic understanding capabilities at extremely low parameter cost. Second, for the transfer from perception to cognition, this application constructs a morphologically-aware visual cues (MVP) mechanism. Unlike coarse masking, the MVP mechanism uses the mask generated by PTF-Net to adaptively generate non-occluded visual cues containing a safety buffer. Combined with our designed soft thinking chain stimulation strategy, this mechanism provides logical guidance to the large medical visual diagnosis model, pointing out the lesion range and providing observation focus, explicitly correcting the attention distribution of LVLM, and ensuring that it outputs a reliable diagnosis based on acquiring non-destructive lesion texture.

[0005] Firstly, this application provides a collaborative analysis method for medical images based on parallel truncated fusion and morphologically-aware visual cues, comprising: A parallel truncated fusion network is constructed to segment acquired medical images and generate segmentation masks. The parallel truncated fusion network includes a visual backbone network, parallel frozen semantic branches of a large language model, and a hybrid gated feature fusion module. The visual backbone network is used to extract visual features from the medical images; the parallel frozen semantic branches of the large language model are used to extract semantic features from the medical images; and the hybrid gated feature fusion module is used to inject the semantic features into the visual backbone network to generate segmentation masks. Adaptive noise filtering and mask restoration are performed on the segmentation mask; based on the restored mask, a context-aware non-occluded bounding box is generated; the safe bounding box is rendered onto the original medical image as a visual anchor point to obtain a visual cue image, and based on the soft cue text, a multimodal input stream containing the visual cue image and text cue is constructed; The multimodal input stream and the acquired clinical questions are input into a large-scale visual language model for medical visual question answering reasoning.

[0006] In one possible design, the frozen semantic branch of the large language model in the parallel setup extracts semantic features from medical images as follows: Visual features obtained from processing the medical images through a visual backbone network are used as input; a truncated index set is defined. For each level index in the collection k Perform the following parallel extraction steps: (1) Through the first k The independent adapters corresponding to the layers perform cross-modal alignment of the visual features: firstly, The input features are flattened into a one-dimensional sequence and then mapped from the channel dimension to the embedding dimension of the large language model through linear projection to obtain the adapted input features. (2) Adapted input features Input to the frozen large language model k The first layer of the Transformer block is calculated using a formula. k semantic features of layers The formula is as follows: (3) Obtain the semantic features of all truncated layers to obtain the semantic feature set. .

[0007] In one possible design, the adapted input features are obtained using the following formula. : In the formula, Indicates input visual features, This means flattening two-dimensional features into a one-dimensional sequence. This represents the linear projection layer of the k-th independent adapter, used to map the visual channel dimension to the embedding dimension of the large language model. This indicates the position code.

[0008] In one possible design, the hybrid gated feature fusion module injects the semantic features into the visual backbone network to generate a segmentation mask in the following manner: Based on visual features The dynamic offset is generated using the following formula: In the formula, w d This represents the dynamic offset, MLP represents the multilayer perceptron, and GAP represents the global average pooling operation. Based on the dynamic offset, the fusion weight is calculated using the following formula: In the formula, This represents the defined learnable static parameter vector. represents the fusion weights, and Softmax represents the normalized exponential function; Based on the fusion weights, a weighted summation operation is performed using the following formula: middle, This represents the semantic features after fusion; Indicates the first k Semantic features output by the LLM layer; This indicates the corresponding fusion weight; Based on the fused semantic features, they are injected into the visual backbone network using the following formula: In the formula, F out Indicates fusion features, Represents the learnable gating coefficients, Adapter out This indicates the output adapter, used for post-processing of unified semantic guided flow features; Based on the fusion features, a low-resolution segmentation mask for the medical image is generated. M low .

[0009] In one possible design, adaptive noise filtering and mask restoration are performed on the segmentation mask, including: The total response area of ​​the foreground pixels in a binary mask is calculated using the following formula: In the formula, S sum Indicates the total response area. i,j This represents the pixel row and column indices of the binary segmentation mask. M i,j This indicates the location of the binary segmentation mask at the pixel position ( i,j The pixel value of ) is 1 for the foreground pixel value and 0 for the background pixel value; when Exceeding the preset significance threshold When a valid lesion is detected, At that time, directly input the original medical image; Low-resolution segmentation mask is obtained by using nearest neighbor interpolation. M low Resample to the original image size to obtain the original size mask. M orig .

[0010] In one possible design, a context-aware, non-occlusion bounding box is generated based on the restored mask using the following formula: In the formula, B safe This represents a context-aware, non-occluded bounding box. x min and x max These represent the original size mask. M orig The minimum and maximum x-coordinates, y min and y max These represent the original size mask. M orig The minimum and maximum ordinates, This represents the safety expansion factor.

[0011] In one possible design, the process of inputting the multimodal input stream along with the acquired clinical questions into a large visual language model for medical visual question answering reasoning is represented as follows: In the formula, A arg max represents the final reasoning answer. Y This indicates the conclusions of all inferences. Y The operation of finding the maximum conditional probability. Represents potential thought chain reasoning paths, The set of all reasonable reasoning paths. X prompt Indicates visual cue images, P ( Y | z , E v ( X prompt ), E t ( Q CoT )) indicates that, given a reasoning path z Visual cues and image-aware context E v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Given the condition, what is the probability that the conclusion is Y? P ( z | Ev ( X prompt ), E t ( Q CoT )) indicates the perceptual context of a given visual cue image. E v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Under the condition of ), the reasoning path is z The probability, Q CoT This indicates a structured thought process prompt.

[0012] Secondly, this application provides a medical image collaborative analysis device based on parallel truncated fusion and morphological perception visual cues, the device comprising: The data fusion module is configured to construct a parallel truncated fusion network to segment the acquired medical images and generate a segmentation mask. The parallel truncated fusion network includes a visual backbone network, parallel frozen large language model semantic branches, and a hybrid gated feature fusion module. The visual backbone network is used to extract visual features from the medical images; the parallel frozen large language model semantic branches are used to extract semantic features from the medical images; and the hybrid gated feature fusion module is used to inject the semantic features into the visual backbone network to generate the segmentation mask. The visual cue module is configured to perform adaptive noise filtering and mask restoration on the segmentation mask; generate a context-aware non-occluded bounding box based on the restored mask; render the safe bounding box as a visual anchor point onto the original medical image to obtain a visual cue image; and construct a multimodal input stream containing the visual cue image and text cue based on the soft cue text. The collaborative reasoning module is configured to input the multimodal input stream and the acquired clinical questions into a large visual language model for medical visual question answering reasoning.

[0013] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the medical image collaborative analysis method based on parallel truncated fusion and morphological perception visual cues as described in the first aspect and various possible designs of the first aspect.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues as described in the first aspect and various possible designs of the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues as described in the first aspect and various possible designs of the first aspect.

[0016] The medical image collaborative analysis method and device based on parallel truncation fusion and morphological perception visual cues provided in this application have at least the following beneficial effects: 1. This application designs a Parallel Truncated Fusion Network (PTF-Net) and proposes a non-destructive semantic injection paradigm. By extracting multi-level features from the frozen LLM in parallel and designing an efficient feature fusion module to replace complex dynamic routing, we successfully solve the feature oversmoothing problem in existing serial methods. Experiments show that this architecture significantly improves the accuracy of segmentation boundaries with only a 0.59M (out of a total of 1.98M) increase in trainable parameters compared to the original MobileU-ViT architecture.

[0017] 2. This application designs a morphological perception visual cueing mechanism (MVP). Specifically, addressing the shortcomings of existing localization and annotation methods where masking obscures lesion details, a collaborative strategy of non-occluding bounding boxes and soft thought chains is designed. This mechanism constructs visual anchors at the physical level and stimulates the model's evidence-based reasoning ability at the logical level, achieving lossless transfer of perceptual results to cognitive reasoning.

[0018] 3. This application was comprehensively evaluated on dermoscopy (ISIC), breast ultrasound (BUSI, BUS), polypscopy (Kvasir), and some OmniMed VQA dataset tasks. The results show that PTF-Net not only outperforms state-of-the-art models, including MobileU-Vit and LLM4Seg, in segmentation metrics, but also significantly surpasses the baseline in VQA tasks. On the ISIC2019 and ISIC2020 VQA tasks, the accuracy was improved by an average of 1.21% and 4.62%, respectively, demonstrating the great potential of this application in practical applications. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A structural diagram of a morphology-aware collaborative analysis framework provided in an embodiment of this application; Figure 2 A flowchart illustrating a medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues, provided for embodiments of this application; Figure 3 An architecture diagram of the parallel truncated fusion module (PTF-Net) provided for embodiments of this application; Figure 4 This is a form-aware visual cue guidance diagram provided in the embodiments of this application; Figure 5 A comparison chart of lesion segmentation results between PTF-Net and mainstream segmentation models on multiple medical datasets, provided for embodiments of this application; Figure 6 A comparison of feature activations before and after the Parallel Truncated Fusion (PTF) module in different medical modalities provided in this application embodiment; Figure 7 This is a schematic diagram illustrating the impact of different LLM layers on segmentation performance in an embodiment of this application. Figure 8 This is a structural diagram of a medical image collaborative analysis device based on parallel truncation fusion and morphological perception visual cues provided in an embodiment of this application.

[0021] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0023] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0024] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0025] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0026] Accurate lesion segmentation is the cornerstone of medical image analysis. Early convolutional neural networks (CNNs), such as U-Net and its variants, dominated the field for a long time due to their powerful local texture extraction capabilities. However, they were often limited in their ability to model long-range dependencies when dealing with lesions with complex topological structures or blurred boundaries. The subsequent Transformer architecture introduced a global self-attention mechanism, which led to powerful segmentation models such as TransUNe and Swin-UNet, alleviating this problem to some extent. However, their semantic understanding capabilities are still limited by the availability of limited medical labeled data.

[0027] Recent research has begun to explore using the frozen layers of Large Language Models (LLMs) as a general knowledge base to enhance visual segmentation. The MedPrompt framework proposed by Sobhan et al. attempts to improve segmentation performance by fusing LLM features with CNN features through a complex dynamic weight routing mechanism. However, these methods typically employ serial embedding or deep coupling strategies, forcing high-resolution visual features to flow through low spatial resolution LLM layers. This not only introduces huge computational overhead but also leads to severe oversmoothing of spatial features, causing the loss of high-frequency diagnostic features such as spiculations and lobulations at lesion edges.

[0028] Large-scale visual-language models (LVLMs) have shown great potential in medical question answering, but factual illusions—descriptions generated by the model that do not match the facts of the image—remain a core bottleneck hindering their clinical application. Existing solutions mainly fall into two categories: external knowledge augmentation and internal perception calibration.

[0029] In terms of external knowledge enhancement, the Retrieval Augmentation (RAG) paradigm aims to suppress randomness in the generation process by introducing reliable clinical evidence. Research has evolved from early text retrieval to multimodal retrieval and knowledge graph reasoning. These works essentially assist model decision-making by improving the signal-to-noise ratio of input information, but they focus on addressing errors caused by knowledge scarcity, rather than correcting endogenous hallucinations stemming from the model's own perceptual deficiencies. In contrast, research on the model's internal perceptual mechanisms has revealed the deeper causes of hallucinations. The paper "Li Y, Du Y, Zhou K, et al. Evaluating object hallucination in large vision-language models[J]. arXivpreprint arXiv:2305.10355, 2023" points out that visual uncertainty leads to guessing when feature extraction is hindered, which is particularly serious in medical scenarios where lesions are highly homogeneous. The paper "Yin S, Fu C, Zhao S, et al. A survey on multimodal large language models[J]. National Science Review, 2024, 11(12): nwae403" further attributes this to modal laziness, meaning that the model tends to ignore the image when the visual signal is weak, degenerating into a statistical shortcut that relies on language priors. Nguyen et al. confirmed the harm of localization bias, that is, the drift of attention to the background region directly leads to factual errors. To address the above shortcomings, existing work has focused on back-end decoding strategies. Although Woodpecker's post-processing text correction or Med-VCD's visual contrast decoding can effectively suppress language priors, both require expensive iterative computation, which significantly increases inference latency.

[0030] Visual cues, as a mechanism for explicitly guiding model attention, effectively compensate for the perceptual deficiencies of general-purpose large models in locating small lesions by superimposing specific markers at the input image level. Previous studies have confirmed that simple cues such as red circles can reshape the attention distribution of the visual encoder, focusing it on the labeled region. In medical image segmentation tasks, sparse geometric cues have also been shown to significantly reduce the difficulty of organ segmentation. Furthermore, research in the field of Med-VQA (Medical Visual Question Answering) has further demonstrated that appropriate visual guidance can effectively suppress the generation of hallucinations in complex backgrounds and improve diagnostic accuracy.

[0031] However, existing technologies often employ static, manually defined prompts and neglect the crucial contextual effects in medical imaging diagnosis. Masks or tight bounding boxes directly covering lesions can easily obscure the infiltrative zone. Radiomics studies indicate that the tumor microenvironment contains key biomarkers for predicting benign or malignant tumors, and disrupting its visual integrity can lead to diagnostic bias.

[0032] Based on the above research on existing technologies, it is known that existing medical multimodal large models often lack fine-grained visual perception capabilities when processing high-resolution images, leading to attention drift and even factual hallucinations. Furthermore, dedicated medical segmentation networks struggle with deep semantic reasoning. To address these contradictions, this application provides a medical image collaborative analysis method based on parallel truncated fusion and morphologically-aware visual cues. By designing a morphologically-aware collaborative diagnosis framework, a decoupled approach is adopted—first precisely locating lesions and then guiding the medical large model's reasoning—to achieve deep collaboration between the two. First, in the perception stage, this paper proposes a parallel truncated fusion network (PTF-Net). This network abandons the serial embedding paradigm, which easily leads to the loss of high-frequency spatial information, and adopts a non-destructive parallel two-stream architecture. It uses the sparse hierarchical semantics of the frozen large language model (LLM) as residual supplementation, injecting it into the visual backbone through a hybrid gating mechanism. Subsequently, in the cognition stage, a morphologically-aware visual cues (MVP) mechanism is constructed. We transform the high-precision segmentation mask generated by the segmentation network into morphology-aware visual cues. These cues are then used to guide the downstream LVLM to focus on the lesion region through non-occluded bounding boxes and thought chains. This mechanism provides visual anchors while fully preserving the crucial lesion periphery features for diagnosis, effectively avoiding the information occlusion risk of traditional masks. Experiments on ISIC2018, BUSI, BUS, Kvasir, and some OmniMed VQA datasets demonstrate that PTF-Net outperforms current mainstream methods in segmentation tasks, achieving an average IoU improvement of 0.53% compared to the baseline across the four datasets. Furthermore, the MVP mechanism significantly suppresses diagnostic illusions in large models, improving the accuracy of medical VQA tasks and providing a new approach for cross-modal collaborative diagnosis.

[0033] like Figure 1The diagram shown illustrates the structure of a morphological perception collaborative analysis framework proposed in this application. The core design of this framework mimics the cognitive logic in medical image analysis, functionally dividing the complex task of medical image understanding into two interconnected stages: high-precision lesion region segmentation and guided visual semantic reasoning. Through the decoupling and collaboration of perception and cognition, an interpretable image analysis path is constructed, from locating visual targets to understanding their semantic information. The front end uses a parallel truncated fusion network (PTF-Net) as the perception center to segment medical images, while the back end processes the segmented target region information and then guides a large medical model to perform visual question answering (VQA) or description generation.

[0034] Please combine further Figure 2 As shown, the medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues specifically includes the following steps S10-S30.

[0035] S10: Construct a parallel truncated fusion network to segment the acquired medical images and generate a segmentation mask. The parallel truncated fusion network includes a visual backbone network, a parallel-configured frozen semantic branch of a large language model, and a hybrid gated feature fusion module. The visual backbone network is used to extract visual features of the medical images; the parallel-configured frozen semantic branch of the large language model is used to extract semantic features of the medical images; and the hybrid gated feature fusion module is used to inject semantic features into the visual backbone network to generate a segmentation mask.

[0036] In this embodiment, the Parallel Truncated Fusion Network (PTF-Net) serves as the perceptual hub of the framework. PTF-Net aims to address the limitations of traditional lightweight CNNs in extracting global semantics. Unlike recent studies that employ a serial embedding strategy that forces high-resolution visual features through low-resolution LLM layers, this embodiment designs a non-destructive parallel dual-stream architecture. The backbone of this architecture uses a lightweight MobileU-Vit network to ensure real-time inference speed in clinical settings; while the bypass introduces a large language model (LLM) with frozen parameters as an external semantic knowledge base. The visual backbone network focuses on extracting high-frequency spatial information such as edges and textures, while the parallel LLM branches focus on extracting abstract medical semantic features. The two paths do not interfere with each other during the feature extraction stage, and information fusion only occurs at specific fusion nodes through lateral connections. This parallel design ensures the spatial fidelity of the original image and largely solves the problem of feature oversmoothing in serial architectures.

[0037] In one exemplary implementation, such as Figure 3As shown, in the semantic extraction branch, this embodiment does not allow the image token to flow completely through all layers of the LLM. Instead, based on the sparsity assumption of medical semantics, a truncation extraction strategy is adopted. The semantics of medical images are often distributed at a specific depth. Layers that are too shallow contain too much general texture, while layers that are too deep may contain purely linguistic abstractions unrelated to vision. Therefore, this embodiment selectively truncates and extracts specific intermediate layers of the LLM.

[0038] The parallel-configured frozen semantic branch of the large language model extracts semantic features from medical images as follows: Visual features obtained from the medical image processed by the visual backbone network are used as input; a truncated index set is defined. For each level index in the collection k Perform the following parallel extraction steps: (1) Through the first k The independent adapters corresponding to the layers perform cross-modal alignment of the visual features: firstly, The input features are flattened into a one-dimensional sequence and then mapped from the channel dimension to the embedding dimension of the large language model through linear projection to obtain the adapted input features. : In the formula, Indicates input visual features, This means flattening two-dimensional features into a one-dimensional sequence. This represents the linear projection layer of the k-th independent adapter, used to map the visual channel dimension to the embedding dimension of the large language model. This indicates the position code.

[0039] (2) Adapted input features Input to the frozen large language model k The first layer of the Transformer block is calculated using a formula. k semantic features of layers The formula is as follows: (3) Obtain the semantic features of all truncated layers to obtain the semantic feature set. .

[0040] In one exemplary implementation, such as Figure 3 As shown, in order to efficiently inject the adapted multi-level LLM semantics into the visual backbone, this embodiment proposes a feature fusion module based on a hybrid gating mechanism. This module introduces a dual adjustment strategy of static benchmark and dynamic offset to balance general knowledge priors and the adaptability of specific samples.

[0041] The core idea of ​​this module is to integrate rather than filter multi-granularity features. Specifically, it involves fusion weights. The generation process first defines a learnable static parameter vector. This is used to capture global statistical priors of hierarchical importance. Simultaneously, it utilizes visual features of the input... Perform global average pooling (GAP) and multilayer perceptron (MLP) mapping to generate dynamic offsets that are highly correlated with the content of the current sample. The final fusion weight is obtained by superimposing the two weights and then normalizing them using Softmax: This mechanism ensures that the model maintains training stability while dynamically adjusting the dependency weights on LLM semantics based on the complexity of the image content. After obtaining the hybrid gating weights (i.e., the final fusion weights), the hybrid gating feature fusion module performs a weighted summation operation: In the formula, This represents the semantic features after fusion; Indicates the first k Semantic features output by the LLM layer; This indicates the corresponding fusion weight.

[0042] To inject semantics without compromising the spatial fidelity of the original visual features, this embodiment employs residual connections for fusion. The fused semantic feature stream is processed by an output adapter and then passed through a learnable gating coefficient. Injected into the visual backbone network, that is: In the formula, F out Indicates fusion features, Represents the learnable gating coefficients, Adapter out This indicates the output adapter, used for post-processing of unified semantic guided flow features.

[0043] Based on fusion features F out Generating low-resolution segmentation masks for medical images M low Specifically, based on the output features after fusing visual and semantic streams. F outThe image is then fed into a lightweight segmentation decoder of PTF-Net. The decoder gradually restores the feature space dimension through upsampling operations and further refines the lesion edge features using convolutional layers. Subsequently, a lesion probability map matching the size of the input low-resolution medical image is generated through the last convolutional layer. The probability map is then binarized using a preset threshold to distinguish the lesion foreground from the normal tissue background, ultimately generating a low-resolution segmentation mask. M low .

[0044] This injection-based design preserves high-frequency texture details in the visual backbone while intelligently introducing deep language priors through a hybrid gating mechanism, significantly improving the model's ability to perceive fuzzy boundaries and complex pathological structures.

[0045] S20: Perform adaptive noise filtering and mask restoration on the segmentation mask; generate a context-aware non-occluded bounding box based on the restored mask; render the safe bounding box as a visual anchor point onto the original medical image to obtain a visual cue image, and construct a multimodal input stream containing the visual cue image and text cue based on the soft cue text.

[0046] In one exemplary implementation, combined with Figure 4 As shown, adaptive noise filtering and mask restoration are achieved in the following way: Shape-aware visual cue generator The primary task is to purify the lesion region and construct a robust feature alignment mechanism. Since the probability map output by the segmentation network may be affected by random perturbations, producing minute false positive noise, directly generating cue boxes based on these noises could easily mislead the LVLM to focus on irrelevant regions. Therefore, this embodiment designs an area-based adaptive filtering strategy. The system first calculates the total response area of ​​the foreground pixels in the binary mask. S sum : In the formula, S sum Indicates the total response area. i,j This represents the pixel row and column indices of the binary segmentation mask. M i,j This indicates the location of the binary segmentation mask at the pixel position ( i,j The pixel value of ) is 1 for the foreground and 0 for the background.

[0047] Only when Exceeding the preset significance threshold (In this experiment, it was empirically verified that the resolution was set to 50 pixels) before the system recognized a valid lesion; otherwise, if... The system will automatically trigger a robust rollback mechanism, directly abandoning visual cues and inputting the original full image. This design establishes a bottom line of "better to have none than a bad one," effectively avoiding cognitive biases introduced by empty or noisy boxes. For valid masks that pass verification, this embodiment further performs connected component analysis and, based on the clinical principle of diagnosing the primary lesion, selects the connected component with the largest area. As the target object, the core diagnostic area can be located in the context of multiple lesions or complex backgrounds.

[0048] To balance inference speed and feature accuracy, the front-end PTF-Net typically runs at low resolution, while the back-end VQA must rely on the original high-resolution image to capture subtle pathological textures. To eliminate this scale difference, this embodiment abandons simple coordinate scaling and instead employs nearest-neighbor interpolation to mask the low-resolution binary image. Resample back to the original image size : In the formula, M orig Upsample represents the original size mask. NN This indicates the nearest neighbor interpolation upsampling operation, where H and W represent the height and width of the original medical image, respectively.

[0049] This interpolation method avoids the grayscale smoothing introduced by bilinear interpolation, maintains the sharpness of the binary mask edges, and ensures that the subsequently generated geometric coordinates are strictly aligned with the original anatomical structure.

[0050] In one exemplary embodiment, considering that traditional masking or tight bounding boxes often obscure the infiltrative zone at the lesion edge, thus compromising a key basis for benign or malignant differentiation, this embodiment employs a context-aware, non-occluding bounding box at the geometric boundary of the largest connected region to transform irregular lesion contours into visually understandable symbols for LVLM without destroying key pathological features within and around the lesion. Based on this, a safety expansion factor was introduced. The context-aware, non-occlusion bounding box is represented as: In the formula, B safe This represents a context-aware, non-occluded bounding box. x min and x max These represent the original size mask. M orig The minimum and maximum x-coordinates, y min and ymax These represent the original size mask. M orig The minimum and maximum ordinates.

[0051] This significant expansion operation is equivalent to establishing a semantic buffer zone around the lesion, ensuring that edge features such as spurs and satellite lesions are fully contained within the field of view. At the visualization level, this bounding box is rendered as a red rectangle with complete whitespace inside. This design explicitly marks the ROI while completely avoiding the risk of pixel occlusion, ensuring that the LVLM receives lossless, original texture information. From a signal processing perspective, the high-frequency color signals of the red box constitute a strong gradient response in the feature space, acting as a visual anchor point and effectively capturing the Transformer's attention head.

[0052] S30: Input the multimodal input stream and the acquired clinical questions into a large-scale visual language model for medical visual question answering reasoning.

[0053] In one exemplary implementation, in order to suppress the intuitive illusions common in medical diagnosis using generalized large models and to improve interpretability, this embodiment adopts the baseline model-specific structured instructions ( <think> ...< / think> This mechanism acts as a trigger, forcing the model to switch from a fast, intuitive mode to a rational, slow-thinking mode. It requires the model to first generate an intermediate reasoning path, explicitly describing the visual evidence within the red box, before generating a final diagnosis. Ultimately, the mathematical essence of VQA answer generation is remodeled within a specific visual perception context. and the constraints of structured thinking chains The following is a conditional probability maximization problem: In the formula, A arg max represents the final reasoning answer. Y This indicates the conclusions of all inferences. Y The operation of finding the maximum conditional probability. Represents potential thought chain reasoning paths, The set of all reasonable reasoning paths. X prompt Indicates visual cue images, P ( Y | z , E v ( X prompt ), E t ( Q CoT )) indicates that, given a reasoning path z Visual cues and image-aware contextE v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Given the condition, what is the probability that the conclusion is Y? P ( z | E v ( X prompt ), E t ( Q CoT )) indicates the perceptual context of a given visual cue image. E v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Under the condition of ), the reasoning path is z The probability, Q CoT This indicates a structured thought process prompt.

[0054] In conclusion, Figure 1 In the architecture shown, the front-end PTF-Net is responsible for solving the geometric localization problem of lesion location, while the back-end LVLM is responsible for solving the semantic reasoning problem of lesion details, effectively balancing the introduction of localization information with the freedom of large-scale model reasoning. This not only significantly improves the accuracy of VQA but also endows the AI ​​system with strong interpretability, because medical staff can not only see the final diagnostic conclusion but also intuitively confirm which anatomical region the AI ​​made the judgment based on through the red box. If the red box is incorrectly located, the doctor can immediately identify and intervene. This feature greatly establishes a trust mechanism in human-machine collaboration and is a key advantage that distinguishes this framework from traditional black-box models.

[0055] To further illustrate the feasibility and progressiveness of the method proposed in this application, a detailed explanation will be provided below through specific experiments.

[0056] Datasets: To verify the segmentation capabilities and robustness of PTF-Net across different medical modalities, this embodiment selects four publicly available medical image segmentation datasets for evaluation, covering ultrasound, dermoscopy, and endoscopic images. Breast ultrasound datasets BUSI and BUS are used to evaluate the model's ability to segment lesions in low-contrast and blurred boundary conditions. Dermoscopy dataset ISIC contains rich texture features, used to test the model's ability to capture pigment networks and texture details. Polypscopy dataset Kvasir is used to evaluate segmentation performance in endoscopic scenarios. In this embodiment, the BUSI, BUS, and ISIC datasets are randomly split into training and validation sets in a 7 / 3 ratio. For the Kvasir dataset, this embodiment follows the official training / validation split criteria and additionally splits a separate test set to ensure the fairness of the evaluation. For the downstream visual question answering (VQA) task, this embodiment selects subsets of ISIC2019 and ISIC2020 from the OmniMedVQA dataset to construct an evaluation benchmark, focusing on testing the model's accuracy in pathological analysis during the "localization followed by diagnosis" task.

[0057] Implementation Details: All experiments in this embodiment are implemented using the PyTorch framework and trained and inferred on a single NVIDIA GeForce RTX 4070 GPU. PTF-Net uses the lightweight MobileU-ViT as the visual backbone to ensure efficient feature extraction. The parallel semantic branch loads the weights of layers 17 and 27 of DeepSeek-R1-Distill-Qwen-1.5B by default. During training, the LLM branch and its corresponding Transformer layer remain completely frozen, and only the visual backbone, feature fusion module (FFM), and lightweight projection layer are trained. The input image resolution is uniformly adjusted to 256x256. For fair comparison, this embodiment adopts the data augmentation strategy of previous 2D segmentation methods. Benchmark methods and evaluation metrics: This embodiment compares PTF-Net with a variety of state-of-the-art segmentation methods, including: U-Net, UNeXt

[30] , TinyU-Net, nnunet, CMU-Net

[33] , Swin-Unet, UCTransNet, MobileU-ViT, UniRepLKNet, LLM4Seg, EGE-Unet, and ERDUnet.

[0058] It should be noted that the specific sources of the above segmentation methods are as follows: U-Net:Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutionalnetworks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer(2015); UNeXt:Jeya Maria Jose Valanarasu and Vishal M Patel. 2022. Unext:Mlp-based rapid medical image segmentation network. In MICCAI. Springer, 23–33; TinyU-Net:Junren Chen, Rui Chen, Wei Wang, Junlong Cheng, Lei Zhang,and Liangyin Chen. 2024. TinyU-Net: Lighter Yet Better U-Net with CascadedMulti-receptive Fields. In MICCAI. Springer, 626–635; nnunet:Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-basedbiomedical image segmentation. Nature methods 18(2), 203–211 (2021); CMU-Net:Fenghe Tang, Lingtao Wang, Chunping Ning, Min Xian, andJianrui Ding. 2023. CMU-Net: a strong ConvMixer-based medical ultrasoundimage segmentation network. In 2023 IEEE 20th International Symposium onBiomedical Imaging (ISBI). IEEE, 1–5; Swin-Unet,UCTransNet:Haonan Wang, Peng Cao, Jiaqi Wang, and Osmar RZaiane. 2022. Uctransnet: rethinking the skip connections in u-net from achannel-wise perspective with transformer. In Proceedings oftheAAAIconference on artificial intelligence, Vol. 36; MobileU-ViT:Tang F, Nian B, Ding J, et al. Mobile U-ViT: Revisitinglarge kernel and U-shaped ViT for efficient medical image segmentation[C] / / Proceedings of the 33rd ACM International Conference on Multimedia. 2025:3408-3417; UniRepLKNet:Xiaohan Ding, Yiyuan Zhang, Yixiao Ge, Sijie Zhao, LinSong, Xiangyu Yue, and Ying Shan. 2024. Unireplknet: A universal perceptionlarge-kernel convnet for audio video point cloud time-series and imagerecognition. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition. 5513–5524; LLM4Seg: Tang F, Ma W, He Z, et al. Pre-trained llm is a semantic-aware and generalizable segmentation booster[C] / / International Conference onMedical Image Computing and Computer-Assisted Intervention. Cham: SpringerNature Switzerland, 2025: 402-412; EGE-Unet: Jiacheng Ruan, Mingye Xie, Jingsheng Gao, Ting Liu, andYuzhuo Fu. 2023. EGE-UNet: an Efficient Group Enhanced UNet for skin lesionsegmentation. In MICCAI. Springer, 481–490; ERDUnet: Hao Li, Di-Hua Zhai, and Yuanqing Xia. 2023. ERDUnet: Anefficient residual double-coding unet for medical image segmentation. TSCVT34, 4 (2023), 2083 -2096.

[0059] This embodiment uses IOU and F1 scoring for BUSI, BUS, ISIC, and Kvasir. Furthermore, this embodiment reports the number of parameters (M) and the number of floating-point operations (GFLOPs) to evaluate the computational complexity of the model.

[0060] Segmentation Task: Table 1 presents a systematic comparison of PTF-Net with 10 current mainstream segmentation paradigms on four medical datasets. Overall, PTF-Net establishes a new performance benchmark while maintaining an extremely low parameter count (1.98M). It not only performs robustly on common datasets such as BUS and Kvasir, but also achieves a significant advantage in average IoU. Compared to CMU-Net with a parameter count as high as 49.93M and nnU-Net with 26.10M, the method in this embodiment achieves equal or even better segmentation accuracy with less than 1 / 20th of their computational cost.

[0061] Table 1: Results on a two-dimensional medical dataset. Best results are highlighted.

[0062] A deeper analysis of the model's performance in challenging scenarios reveals more about PTF-Net's robustness. For example... Figure 5 As shown, in the ISIC2018 (dermoscopy) dataset, which is rich in texture features and extremely complex, traditional methods such as UNeXt and UniRepLKNet are often hampered by the complex pigment network interference around lesions, making it difficult to accurately define boundaries. However, PTF-Net achieved an IoU of 83.52% on this dataset, significantly outperforming all comparable methods. This overwhelming advantage stems from the fact that the model does not simply rely on local texture features, but successfully injects global semantic priors through frozen LLM layers, thereby achieving accurate capture of irregular lesion edges in pixel-level classification. Similarly, on the BUSI (breast ultrasound) dataset, which is full of artifact interference and low contrast, PTF-Net still maintains an IoU of 72.92%, outperforming the similar lightweight model TinyU-Net (66.21%) and the large model-enhanced LLM4Seg (71.91%). This is thanks to the parallel truncation fusion mechanism, which allows the model to "borrow" the high-level semantics of LLM to logically complete the lesion morphology when visual features are blurred, rather than blindly guessing.

[0063] Finally, as Figure 6 As shown, this embodiment further verifies the superiority of its architecture design through a deep comparison with its competitor, LLM4Seg. Although both use DeepSeek as an external knowledge source, LLM4Seg's serial structure forces high-resolution visual features through a low-rank linguistic space, resulting in the irreversible loss of high-frequency information. Experimental data clearly reflect this: on the ISIC and BUSI datasets, which have extremely high detail requirements, PTF-Net's IoU is 0.89% and 1.01% higher than LLM4Seg, respectively. This result strongly supports the core argument of the parallel non-destructive injection strategy proposed in this embodiment, namely, by placing LLM in the bypass and performing multi-level feature fusion, the model can significantly enhance semantics while preserving high-frequency details of the visual backbone.

[0064] VQA Task: To verify the practical effectiveness of the "localization first, diagnosis later" framework in clinical decision-making, this embodiment constructs a rigorous cross-domain reasoning evaluation experiment based on the OmniMedVQA benchmark. Specifically, this embodiment sets the front-end PTF-Net segmentation model to be trained only on the ISIC 2018 dataset, while the downstream VQA diagnostic task is performed on the ISIC 2019 / 2020 datasets across multiple years, aiming to test the effectiveness of the transformation from front-end perceptual priors to back-end cognitive reasoning. In the dermoscopy (Der) task, which is from the same source as the training data, the experimental results show that the method of this embodiment exhibits excellent performance ceiling and stability. On the ISIC 2019 test set (Table 2), the method of this embodiment achieves a high accuracy of 92.77%, surpassing the baseline model Med-R1 2B's 92.52%. On the more challenging ISIC 2020 test set (Table 3), the method of this embodiment also maintains the high level of the baseline model at 96.31%. Although the numerical improvement in the high-score range may seem small, it represents a key breakthrough in overcoming the long-tail problem. Qualitative analysis shows that traditional VLMs are prone to producing factual illusions due to attentional drift when faced with complex samples containing surgical markers, dense hair, or measurement scales. In contrast, the model in this embodiment, thanks to the high-precision morphological mask provided by the front-end PTF-Net, successfully constructs mandatory visual anchors, physically filtering out background noise and locking attention onto the lesion region (ROI), thereby ensuring the rigor of the diagnostic logic and the consistency of visual facts.

[0065] Meanwhile, the results in Tables 2 and 3 profoundly reveal the significant compensatory effect of front-end segmentation priors in cross-modal cognitive transfer. Even when the large back-end model is trained on modalities completely heterogeneous with dermatopathology, such as CT, MRI, or X-ray, and faces severe domain shift challenges, the introduction of front-end morphological perception visual cues (MVP) still significantly enhances its inference robustness on the ISIC dermoscopy test set. In Table 2, taking the CT-trained model that has never seen skin images as an example, its diagnostic accuracy increased from 76.56% to 78.55% after introducing the segmentation guidance of this embodiment. The results in Table 3 further demonstrate the universality of this conclusion. In the ISIC2020 VQA task, the accuracy of the MRI-trained model improved by 8 percentage points after introducing MVP, and in the US modality model with weaker basic cognitive abilities, the method of this embodiment further boosted its accuracy from a failing 54.15% to 64.00%. These significant gains indicate that, despite the lack of domain-specific pathological knowledge in the cognitive model, the high-precision spatial localization provided by the front-end network can act as a physical anchor, forcing the model to focus its attention from irrelevant background noise to the lesion area, thereby effectively bridging the cognitive gap.

[0066] This phenomenon of strong perception compensating for weak cognition has also been consistently verified in other heterogeneous modalities such as fundus imaging (FP) and X-ray imaging. Overall, after introducing the MVP mechanism, the model's overall average accuracy improved to 80.92% on ISIC 2019 and to 80.08% on ISIC 2020. This strongly demonstrates that the decoupled collaborative framework proposed in this embodiment has extremely high generalization potential; that is, through explicit morphological perception constraints, the model can utilize general saliency localization capabilities for logical calibration when cognitive abilities are limited or when facing tasks in unknown domains. This not only verifies the effectiveness of the clinical intuition of "see clearly first, then understand," but also provides a practical technical path for building a low-illusion, highly robust AI assistant capable of handling open-world medical scenarios.

[0067] Table 2: Performance of Med-R1 trained in eight different modalities and Med-R1 guided by front-end segmentation on the ISIC2019 test set. All values ​​represent accuracy; bold indicates higher accuracy for the corresponding training-test pair in each column.

[0068] Table 3: Performance of Med-R1 trained in eight different modalities and Med-R1 guided by front-end segmentation on the ISIC2020 test set. All values ​​represent accuracy; bold indicates higher accuracy for the corresponding training-test pair in each column.

[0069] To delve into the underlying mechanisms of PTF-Net's performance improvement and verify the effectiveness of each core component, this embodiment designed multiple sets of controlled variable experiments on the BUS and ISIC datasets. This embodiment focuses on discussing the impact of architectural paradigm, semantic hierarchy selection, and visual cueing strategies on the final performance.

[0070] Ablation Study of Parallel Fusion Architecture: First, this embodiment explores the impact of the LLM semantic injection architecture on segmentation performance. To verify the superiority of the PTF architecture, this embodiment constructs three variant models for comparison: (1) Serial Embedding Baseline: Following methods such as LLM4Seg, the LLM layer is directly inserted between the encoder and decoder; (2) Naive Parallel Fusion: The parallel dual-stream architecture of this embodiment is adopted, but only simple element-wise addition is used to replace FFM; (3) Complete PTF-Net.

[0071] Experimental results show that the serial architecture suffers from significant feature bottlenecks. Although semantic priors are introduced, the IoU improvement of the serial baseline compared to the pure visual backbone is limited on the BUS dataset. This is because high-resolution visual features are forced to pass through a low-resolution LLM space, resulting in the irreversible loss of high-frequency details such as edge spikes. In contrast, the parallel architecture preserves the complete spatial information flow through bypass branches, significantly alleviating the feature oversmoothing problem.

[0072] Further comparison of fusion strategies reveals that naive summation fusion cannot effectively handle cross-modal distribution differences and even introduces semantic noise. The hybrid gated feature fusion module proposed in this embodiment, however, adaptively calibrates the weights of general knowledge and specific sample features through a dynamic gating mechanism. When dealing with samples with severe artifacts in the BUS dataset, FFM automatically increases the weight of semantic flow to assist shape completion; while in samples with clear textures, it emphasizes visual flow. This dynamic adaptation mechanism brings robust improvements in IoU and F1 performance to the model.

[0073] Table 4: Ablation of Additional Layers on Large Models

[0074] Semantic Hierarchy Selection and Sparse Hierarchy Hypothesis: To verify the sparse hierarchy hypothesis of medical image semantics and explore the optimal feature injection strategy, this embodiment conducts detailed ablation analysis on different combinations of LLM truncated layers on the BUS dataset. Experiments compare the performance differences of the baseline model, single-layer injection, and the multi-layer parallel fusion strategy proposed in this embodiment. Specific results are as follows: Figure 7 As shown.

[0075] First, this embodiment evaluates the impact of introducing a specific LLM layer alone on segmentation performance. Quantitative data shows that introducing pre-trained semantics of any depth improves baseline performance. Specifically, model performance peaks at layer 17, then slightly declines and plateaus from layers 18 to 27. This trend is largely consistent with observations of LLM4Seg, where medical semantic representations in LLMs are often richest in the middle of the network. However, relying solely on the strongest single layer exhibits significant performance saturation. Although layer 17 shows a significant improvement over the baseline, it is limited to single-dimensional feature representation, making it difficult to capture the complex texture details and global topological structure of medical images.

[0076] The core of this embodiment lies in leveraging the significant functional complementarity of LLM layers of different depths. Mid-level LLM layers excel at capturing high-frequency textures and echo heterogeneity, while deeper LLM layers are more robust to global anatomical shapes. Based on this, PTF-Net does not limit itself to selecting a single optimal layer, but instead fuses layers 17 and 27 through a parallel architecture. This strategy is not a simple feature overlay, but rather constructs dual constraints on texture and shape. Experimental results confirm that the dual-layer fusion strategy further breaks through the performance ceiling of a single layer (17), demonstrating the strongest generalization ability. This finding strongly proves that the PTF architecture, by selectively extracting complementary layers, maximizes semantic value while effectively avoiding computational redundancy caused by introducing all layers, achieving the optimal balance between accuracy and efficiency.

[0077] Ablation study of morphological perception visual cue strategy: This embodiment also evaluates the impact of different visual cue forms on downstream VQA diagnostic tasks. This embodiment sets the experiment with three conditions: (1) Baseline: no visual cue, only the original image is input; (2) Masking: the lesion area is covered with a semi-transparent mask; (3) MVP: the non-masking safety bounding box proposed in this embodiment.

[0078] The experimental results are shown in Table 2. Compared to the baseline, introducing visual cues improves diagnostic accuracy in most modalities, demonstrating that addressing attentional drift is a prerequisite for improving the clinical performance of LVLM. However, the traditional masking strategy even leads to performance degradation in some modalities (such as US and Micro), and its overall accuracy is lower than the MVP method proposed in this embodiment. In particular, the MVP strategy shows significant advantages in fundus photography (FP) and CT modalities.

[0079] Qualitative analysis revealed that the fundamental reason for the performance difference lay in the fact that the mask directly covered the infiltration zone inside and at the edges of the lesion, causing LVLM to lose the texture details crucial for determining benignity or malignancy. In contrast, the MVP mechanism, through a safe bounding box, explicitly focused attention while fully preserving the pathological features surrounding the lesion. Combined with soft cues and a structured thought process, this strategy not only achieved the highest diagnostic accuracy but also effectively suppressed factual illusions, ensuring that the diagnostic logic was derived from the complete anatomical context.

[0080] The morphological perception-based collaborative diagnosis and treatment framework proposed in this embodiment demonstrates excellent performance in medical image analysis. These empirical results not only verify the effectiveness of the architecture but also provide strong experimental support for the two core designs proposed in this embodiment. Accordingly, this section will explore these findings in depth from two dimensions: the theoretical basis of the architecture design and the clinical value of perception-cognition decoupling.

[0081] Effectiveness of Parallel Fusion Architecture: In medical image segmentation tasks, the core challenge lies in balancing the preservation of high-resolution spatial details with the introduction of deep abstract semantics. Traditional large-model fusion schemes often employ a serial embedding paradigm, which forcibly maps visual features to the semantic space of an LLM for processing, and then maps them back to the visual space. This iterative mapping process constitutes a significant information bottleneck. Since the hidden representation of an LLM is essentially a high-level abstraction of textual logic, its spatial sensitivity is far lower than that of a visual encoder. Therefore, when features pass through, high-frequency spatial information such as lesion edge spiculations and microsatellite lesions is easily lost irreversibly, leading to the smoothing of segmentation boundaries. In contrast, the parallel architecture of PTF-Net follows the design principle of non-destructive injection, preserving the complete visual backbone and constructing the frozen LLM as an independent bypass knowledge base. This design concept is mathematically similar to residual learning. In this embodiment, the deep semantics extracted by the LLM are regarded as a high-order residual complement, which is injected into the visual feature stream additively through the feature fusion module.

[0082] Effectiveness of Decoupling Segmentation and Diagnosis: In the inference and diagnosis task, the MVP mechanism in this embodiment demonstrates robustness exceeding the baseline of large medical models, especially in cross-modal scenarios. This result profoundly confirms the necessity of the collaborative approach of localization preceding diagnosis. Experiments in this embodiment show that the safe bounding box generated by the front-end PTF-Net acts as a visual anchor point, forcing the inference space of the back-end LLM to converge to the anatomically relevant region of interest, effectively preventing attention drift. More importantly, this embodiment employs a non-occlusion cueing strategy. Unlike traditional masking, the MVP mechanism fully preserves the infiltration zone at the lesion edge. In pathology, the microenvironment surrounding the tumor contains key biomarkers for predicting benign or malignant tumors. Therefore, the success of the MVP mechanism lies not only in the attention guidance at the algorithmic level but also in its adherence to the pathological diagnostic principle of preserving surrounding features, achieving a deep alignment between AI logic and physician cognitive intuition.

[0083] This embodiment addresses the challenge of balancing perceptual accuracy and inference reliability in medical image analysis by proposing a morphological perception-based collaborative diagnostic framework based on the decoupling of perception and cognition. In the perception phase, PTF-Net employs a non-destructive parallel two-stream design, combined with a hybrid gated feature fusion module (FFM), to inject deep semantic priors from a frozen large language model into the visual backbone via residual supplementation. This mechanism effectively utilizes the sparse hierarchical complementarity of medical semantics while preserving high-frequency texture details of the original image, significantly improving the model's segmentation robustness under blurred boundaries and complex backgrounds. In the cognition phase, the MVP mechanism, through the synergy of non-occluded safe bounding boxes and structured thought chains, fully preserves the lesion's surrounding features while suppressing hallucinations. Experimental results demonstrate that the architecture of this embodiment establishes new performance benchmarks on multiple datasets, validating the core value of explicit morphological guidance in building trustworthy medical AI. Future work will further explore the application potential of this framework in 3D volumetric data and edge deployment.

[0084] This application also provides a medical image collaborative analysis device based on parallel truncation fusion and morphological perception visual cues, such as... Figure 8 As shown, the medical image collaborative analysis device based on parallel truncated fusion and morphological perception visual cues includes: The data fusion module 801 is configured to construct a parallel truncated fusion network to segment the acquired medical image and generate a segmentation mask. The parallel truncated fusion network includes a visual backbone network, parallel frozen large language model semantic branches, and a hybrid gated feature fusion module. The visual backbone network is used to extract visual features from the medical image; the parallel frozen large language model semantic branches are used to extract semantic features from the medical image; and the hybrid gated feature fusion module is used to inject the semantic features into the visual backbone network to generate the segmentation mask. The visual cue module 802 is configured to perform adaptive noise filtering and mask restoration on the segmentation mask; generate a context-aware non-occluded bounding box based on the restored mask; render the safe bounding box as a visual anchor point onto the original medical image to obtain a visual cue image; and construct a multimodal input stream containing the visual cue image and text cue based on the soft cue text. The collaborative reasoning module 803 is configured to input the multimodal input stream and the acquired clinical questions into a large visual language model for medical visual question answering reasoning.

[0085] In some embodiments, the frozen semantic branches of the large language model in the parallel configuration extract semantic features of medical images in the following manner: The visual features obtained from medical images processed by a visual backbone network are used as input; a truncation index set is defined. For each level index in the collection k Perform the following parallel extraction steps: (1) Through the first k The independent adapters corresponding to the layers perform cross-modal alignment of the visual features: firstly, The input features are flattened into a one-dimensional sequence and then mapped from the channel dimension to the embedding dimension of the large language model through linear projection to obtain the adapted input features. ; (2) Adapted input features Input to the frozen large language model k The first layer of the Transformer block is calculated using a formula. k semantic features of layers The formula is as follows: (3) Obtain the semantic features of all truncated layers to obtain the semantic feature set. .

[0086] In some embodiments, the adapted input features are obtained using the following formula. : In the formula, Indicates input visual features, This means flattening two-dimensional features into a one-dimensional sequence. This represents the linear projection layer of the k-th independent adapter, used to map the visual channel dimension to the embedding dimension of the large language model. This indicates the position code.

[0087] In some embodiments, the hybrid gated feature fusion module injects the semantic features into the visual backbone network to generate a segmentation mask in the following manner: Based on visual features The dynamic offset is generated using the following formula: In the formula, w d This represents the dynamic offset, MLP represents the multilayer perceptron, and GAP represents the global average pooling operation. Based on the dynamic offset, the fusion weight is calculated using the following formula: In the formula, This represents the defined learnable static parameter vector. represents the fusion weights, and Softmax represents the normalized exponential function; Based on the fusion weights, a weighted summation operation is performed using the following formula: In the formula, This represents the semantic features after fusion; Indicates the first k Semantic features output by the LLM layer; This indicates the corresponding fusion weight; Based on the fused semantic features, they are injected into the visual backbone network using the following formula: In the formula, F out Indicates fusion features, Represents the learnable gating coefficients, Adapter out This indicates the output adapter, used for post-processing of unified semantic guided flow features; Based on the fusion features, a low-resolution segmentation mask for the medical image is generated. M low .

[0088] In some embodiments, adaptive noise filtering and mask restoration of the segmentation mask include: The total response area of ​​the foreground pixels in a binary mask is calculated using the following formula: In the formula, S sum Indicates the total response area. i,j This represents the pixel row and column indices of the binary segmentation mask. M i,j This indicates the location of the binary segmentation mask at the pixel position ( i,j The pixel value of ) is 1 for the foreground pixel value and 0 for the background pixel value; when Exceeding the preset significance threshold When a valid lesion is detected, At that time, directly input the original medical image; Low-resolution segmentation mask is obtained by using nearest neighbor interpolation. M low Resample to the original image size to obtain the original size mask. M orig .

[0089] In some embodiments, a context-aware, non-occlusion bounding box is generated based on the restored mask using the following formula: In the formula, Bsafe This represents a context-aware, non-occluded bounding box. x min and x max These represent the original size mask. M orig The minimum and maximum x-coordinates, y min and y max These represent the original size mask. M orig The minimum and maximum ordinates, This represents the safety expansion factor.

[0090] In some embodiments, the process of inputting the multimodal input stream and the acquired clinical questions into a large visual language model for medical visual question answering reasoning is represented as follows: In the formula, A arg max represents the final reasoning answer. Y This indicates the conclusions of all inferences. Y The operation of finding the maximum conditional probability. Represents potential thought chain reasoning paths, The set of all reasonable reasoning paths. X prompt Indicates visual cue images, P ( Y | z , E v ( X prompt ), E t ( Q CoT )) indicates that, given a reasoning path z Visual cues and image-aware context E v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Given the condition, what is the probability that the conclusion is Y? P ( z | E v ( X prompt ), E t ( Q CoT)) indicates the perceptual context of a given visual cue image. E v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Under the condition of ), the reasoning path is z The probability, Q CoT This indicates a structured thought process prompt.

[0091] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0092] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0093] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0094] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0095] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues described in the above embodiments.

[0096] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues in the above embodiments.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0098] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0099] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0100] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0101] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0102] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0103] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0104] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0105] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0106] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A collaborative analysis method for medical images based on parallel truncated fusion and morphological visual cues, characterized in that, The method includes: A parallel truncated fusion network is constructed to segment acquired medical images and generate segmentation masks. The parallel truncated fusion network includes a visual backbone network, parallel frozen semantic branches of a large language model, and a hybrid gated feature fusion module. The visual backbone network is used to extract visual features from the medical images; the parallel frozen semantic branches of the large language model are used to extract semantic features from the medical images; and the hybrid gated feature fusion module is used to inject the semantic features into the visual backbone network to generate segmentation masks. Adaptive noise filtering and mask restoration are performed on the segmentation mask; based on the restored mask, a context-aware non-occluded bounding box is generated; the safe bounding box is rendered onto the original medical image as a visual anchor point to obtain a visual cue image, and based on the soft cue text, a multimodal input stream containing the visual cue image and text cue is constructed; The multimodal input stream and the acquired clinical questions are input into a large-scale visual language model for medical visual question answering reasoning.

2. The medical image collaborative analysis method based on parallel truncated fusion and morphological perception visual cues as described in claim 1, characterized in that, The frozen semantic branch of the large language model in the parallel configuration extracts semantic features of medical images in the following way: The visual features obtained by processing the medical images through the visual backbone network are used as input; Define a set of truncated indices For each level index in the collection k Perform the following parallel extraction steps: (1) Through the first k The independent adapters corresponding to the layers perform cross-modal alignment of the visual features: firstly, The input features are flattened into a one-dimensional sequence and then mapped from the channel dimension to the embedding dimension of the large language model through linear projection to obtain the adapted input features. ; (2) Adapted input features Input to the frozen large language model k The first layer of the Transformer block is calculated using a formula. k semantic features of layers The formula is as follows: (3) Obtain the semantic features of all truncated layers to obtain the semantic feature set. .

3. The medical image collaborative analysis method based on parallel truncated fusion and morphological perception visual cues according to claim 2, characterized in that, The adapted input features are obtained using the following formula. : In the formula, Indicates input visual features, This means flattening two-dimensional features into a one-dimensional sequence. This represents the linear projection layer of the k-th independent adapter, used to map the visual channel dimension to the embedding dimension of the large language model. This indicates the position code.

4. The medical image collaborative analysis method based on parallel truncated fusion and morphological perception visual cues according to claim 2, characterized in that, The hybrid gated feature fusion module injects the semantic features into the visual backbone network in the following manner to generate a segmentation mask: Based on visual features The dynamic offset is generated using the following formula: In the formula, w d This represents the dynamic offset, MLP represents the multilayer perceptron, and GAP represents the global average pooling operation. Based on the dynamic offset, the fusion weight is calculated using the following formula: In the formula, This represents the defined learnable static parameter vector. represents the fusion weights, and Softmax represents the normalized exponential function; Based on the fusion weights, a weighted summation operation is performed using the following formula: In the formula, This represents the semantic features after fusion; Indicates the first k Semantic features output by the LLM layer; This indicates the corresponding fusion weight; Based on the fused semantic features, they are injected into the visual backbone network using the following formula: In the formula, F out Indicates fusion features, Represents the learnable gating coefficients, Adapter out This indicates the output adapter, used for post-processing of unified semantic guided flow features; Based on the fusion features, a low-resolution segmentation mask for the medical image is generated. M low .

5. The medical image collaborative analysis method based on parallel truncated fusion and morphological perception visual cues according to claim 1, characterized in that, Adaptive noise filtering and mask restoration of the segmentation mask, including: The total response area of ​​the foreground pixels in a binary mask is calculated using the following formula: In the formula, S sum Indicates the total response area. i,j This represents the pixel row and column indices of the binary segmentation mask. M i,j This indicates the location of the binary segmentation mask at the pixel position ( i,j The pixel value of ) is 1 for the foreground pixel value and 0 for the background pixel value; when Exceeding the preset significance threshold When a valid lesion is detected, At that time, directly input the original medical image; Low-resolution segmentation mask is obtained by using nearest neighbor interpolation. M low Resample to the original image size to obtain the original size mask. M orig .

6. The medical image collaborative analysis method based on parallel truncated fusion and morphological perception visual cues according to claim 5, characterized in that, Based on the restored mask, a context-aware, non-occluded bounding box is generated using the following formula: In the formula, B safe This represents a context-aware, non-occluded bounding box. x min and x max These represent the original size mask. M orig The minimum and maximum x-coordinates, y min and y max These represent the original size mask. M orig The minimum and maximum ordinates, This represents the safety expansion factor.

7. The medical image collaborative analysis method based on parallel truncated fusion and morphological perception visual cues according to any one of claims 1 to 6, characterized in that, The process of inputting the multimodal input stream and the acquired clinical questions into a large-scale visual language model for medical visual question answering reasoning is represented as follows: In the formula, A arg max represents the final reasoning answer. Y This indicates the conclusions of all inferences. Y The operation of finding the maximum conditional probability. Represents potential thought chain reasoning paths, The set of all reasonable reasoning paths. X prompt Indicates visual cue images, P ( Y | z , E v ( X prompt ), E t ( Q CoT )) indicates that, given a reasoning path z Visual cues and image-aware context E v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Given the condition, what is the probability that the conclusion is Y? P ( z | E v ( X prompt ), E t ( Q CoT )) indicates the perceptual context of a given visual cue image. E v ( X prompt ) and structured thinking chain constraints E t ( Q CoT Under the condition of ), the reasoning path is z The probability, Q CoT This indicates a structured thought process prompt.

8. A medical image collaborative analysis device based on parallel truncated fusion and morphological perception visual cues, characterized in that, The device includes: The data fusion module is configured to construct a parallel truncated fusion network to segment the acquired medical images and generate a segmentation mask. The parallel truncated fusion network includes a visual backbone network, parallel frozen large language model semantic branches, and a hybrid gated feature fusion module. The visual backbone network is used to extract visual features from the medical images; the parallel frozen large language model semantic branches are used to extract semantic features from the medical images; and the hybrid gated feature fusion module is used to inject the semantic features into the visual backbone network to generate the segmentation mask. The visual cue module is configured to perform adaptive noise filtering and mask restoration on the segmentation mask; generate a context-aware non-occluded bounding box based on the restored mask; render the safe bounding box as a visual anchor point onto the original medical image to obtain a visual cue image; and construct a multimodal input stream containing the visual cue image and text cue based on the soft cue text. The collaborative reasoning module is configured to input the multimodal input stream and the acquired clinical questions into a large visual language model for medical visual question answering reasoning.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory to implement the medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the medical image collaborative analysis method based on parallel truncation fusion and morphological perception visual cues as described in any one of claims 1-7.

Images