SAM and parameter efficient fine tuning-based self-prompting multi-mode malignant brain tumor segmentation method

Abstract

The invention discloses a self-prompting multi-modal malignant brain tumor segmentation method based on SAM and efficient parameter fine tuning. The method comprises the following steps: acquiring and preprocessing multi-modal medical image data of a target area; based on the preprocessed multi-modal medical image data, a self-prompting multi-modal malignant brain tumor segmentation model is utilized to obtain a segmentation mask of a target area, and the self-prompting multi-modal malignant brain tumor segmentation model comprises a multi-modal feature adaptive fusion module, a lightweight prompt encoder, an SAM and a category converter. The multi-modal feature adaptive fusion module generates corresponding depth image embedding based on the preprocessed multi-modal medical image data, and the lightweight prompt encoder generates prompt embedding for different tumor subregions based on the depth image embedding; based on the depth image embedding and the hint embedding, a mask decoder of the SAM and the category converter generate a multi-category segmentation mask.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing and deep learning, and particularly relates to a self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning. BACKGROUND

[0002] Malignant brain tumors, especially gliomas, are one of the most challenging tumor types in clinical practice. Accurate segmentation of tumor regions from multi-modal magnetic resonance imaging (MRI) is crucial for disease diagnosis, treatment planning, and prognosis. Traditional tumor segmentation algorithms are inefficient and have limited accuracy, while deep learning networks represented by Fully Convolutional Networks (FCN) and nnU-Net have significantly improved the automation and accuracy of segmentation by learning from large-scale datasets.

[0003] In recent years, visual base models pre-trained on massive natural images, such as the Segmentation Anything Model (SAM), have shown strong general segmentation capabilities. However, due to the huge domain gap between natural images and medical images, SAM directly applied to medical tumor segmentation faces serious challenges. To solve this problem, researchers have proposed various fine-tuning strategies, such as MedSAM, SAM-Med2D, etc., proving the great potential of SAM in medical applications.

[0004] However, the existing technology still has the following shortcomings: Insufficient multi-modal information fusion: Existing multi-modal data fusion methods mostly use simple channel concatenation or averaging, which may lead to loss of complementary information between modalities or interference between features; Dependence on artificial prompts: Most fine-tuning methods for SAM still rely on point, box, and other forms of human interaction prompts to guide segmentation, which is inefficient in clinical workflows that require processing of large numbers of slices and is difficult to achieve full automation; Insufficient segmentation accuracy in complex regions: For various sub-regions of tumors, such as enhancing tumors (ET), non-enhancing tumor cores (NETC), surrounding non-enhancing FLAIR hyperintensity (SNFH), and resection cavities (RC), accurate delineation, especially in complex cases after treatment, remains a technical challenge. SUMMARY

[0005] The present application aims at the problems of insufficient multi-modal information fusion, fundamental dependence on artificial prompting, and lack of medical field induction bias in the application of visual basic models to multi-modal medical image segmentation in the prior art, and provides a self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning, which is efficient, accurate and does not require human interaction.

[0006] According to a first aspect of the embodiments of the present application, a self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning is provided, comprising: obtaining multi-modal medical image data of a target region and preprocessing the data; based on the preprocessed multi-modal medical image data, using a self-prompting multi-modal malignant brain tumor segmentation model to obtain a segmentation mask of the target region, the self-prompting multi-modal malignant brain tumor segmentation model comprising a multi-modal feature self-adaptive fusion module, a lightweight prompt encoder, a SAM, and a class converter, the multi-modal feature self-adaptive fusion module generating corresponding deep image embeddings based on the preprocessed multi-modal medical image data, the lightweight prompt encoder generating prompt embeddings for different tumor sub-regions based on the deep image embeddings; based on the deep image embeddings and the prompt embeddings, the mask decoder of the SAM and the class converter generate a multi-class segmentation mask.

[0007] Further, at least one of the multi-modal medical image data is a reference modality that provides the anatomical structure of the target region.

[0008] Further, in the multi-modal feature self-adaptive fusion module: the preprocessed multi-modal medical image data is mapped to two groups according to a predetermined rule or adaptive soft assignment; each group performs primary feature extraction through a parallel channel and inputs the extracted features into a pre-trained SAM image encoder through an input adapter to obtain early embeddings corresponding to the primary features of each group; based on the early embeddings of each group, shared components and group difference components are calculated and reorganized across channels into enhanced representations; based on the enhanced representations, weighted features are refined through a medical heuristic attention mechanism, straight-through features are obtained by processing the preprocessed multi-modal medical image data through a straight-through projection branch, and deep image embeddings are obtained by fusing the weighted features and the straight-through features through a pixel-level gate.

[0009] Further, the medical heuristic attention mechanism is used to weight the enhanced representations by projecting the input enhanced representations into three semantic subspaces corresponding to lesions, boundaries and contexts respectively, and further interacting and enhancing the features of the subspaces through a multi-path gate modulation mechanism, specifically: projecting the original multi-modal input to a lesion subspace, a boundary subspace, and a context subspace; modulating the boundary subspace feature by the lesion subspace feature through a gating composed of a channel gate and a spatial gate to obtain a boundary feature, and modulating the lesion subspace feature by the context subspace feature to obtain a lesion feature; fusing the boundary feature and the lesion feature, and generating an attention map through a multi-scale module and a feature enhancement module, and multiplying the attention map and the enhanced representation element by element to obtain a weighted feature.

[0010] Further, the data processing process of the lightweight prompt encoder includes: The depth image embedding simultaneously enters a local detail channel and a global context channel, and feature extraction is performed in parallel, wherein the local detail channel is used to extract a local detail feature , and the global context channel is used to extract a global feature , wherein is a depth image embedding, , , respectively represent Pointwise convolution, Depthwise convolution, convolution, is group normalization, is an activation function, is an output channel number, is a set of hole rates; The local detail feature and the global feature are dynamically weighted and combined through a spatial gating fusion module to obtain a fused feature; The fused feature is enhanced in spatial context through an efficient anatomical position encoding module, and the enhanced feature is decoded by a multi-category prompt projector into dense prompts for different tumor sub-regions. The dense prompts are encoded by a prompt encoder of the SAM to form a prompt embedding.

[0011] Further, in the efficient anatomical position encoding module: The fused feature is injected with geometric / anatomical priors to generate three parallel position encoding components, namely an anatomical position encoding component , a sinusoidal coordinate position encoding component , and a radial distance position encoding component ; Through an adaptive weight controller, three weights corresponding to the position encoding components are dynamically generated based on the fused feature , and ; weighting the position encoding components with the weights, to obtain position encoding ; injecting the position encoding into the fused feature to obtain an enhanced feature .

[0012] Further, based on the depth image embedding and the prompt embedding, the mask decoder of the SAM and the category converter generate a multi-category segmentation mask, comprising: based on the depth image embedding and the prompt embedding, a segmentation result is generated by using a mask decoder; based on the segmentation result, a multi-category segmentation mask is generated by using the category converter; wherein, in the category converter, the pixel-level feature output by the mask decoder is refined by a combination of convolution, normalization and activation, and is mapped to a plurality of category classes by one layer of convolution, and a multi-category mask is generated by channel dimension Softmax and pixel-by-pixel Argmax.

[0013] Further, the multi-modal feature adaptive fusion module, the lightweight prompt encoder, the category converter and the SAM are all network structures after training, wherein the main parameters of the SAM are kept frozen during training, and a plurality of lightweight adapter parameters are inserted; In the training process, DiceFocalLoss is used as the loss function, which is defined as: , wherein, , is the corresponding loss weight, the Dice loss is used to optimize the area overlap, and the formula is: , wherein is the predicted probability of the model for the pixel , is its corresponding true label, N is the total number of pixels, is a smoothing coefficient to prevent the denominator from being 0; The Focal loss is used to make the model focus on difficult samples, and the formula is: wherein, is the predicted probability of the model for the correct category, is the focusing parameter, is the category weight parameter.​

[0014] According to a second aspect of the embodiments of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect.

[0015] According to a third aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores computer instructions, and the instructions are executed by a processor to implement the steps of the method according to the first aspect.

[0016] The technical solutions provided by the embodiments of the present application can include the following beneficial effects: Efficient multi-modal information fusion: through the MFAM module, the present application can intelligently distinguish and fuse the shared anatomical information and specific pathological features in multi-modal medical images, overcoming the information dilution problem caused by traditional splicing or averaging methods.

[0017] Full-automatic self-prompt segmentation is achieved: through the LPE module, the present application enables the model to generate accurate prompts according to image content, eliminating the dependence on human interaction and realizing an end-to-end automatic segmentation process.

[0018] The segmentation accuracy of complex regions is improved: by introducing modules such as TransMedAttention with medical induction bias, the present application can better understand and distinguish clinical concepts such as lesions, boundaries, and context, significantly improving the segmentation accuracy and robustness of malignant brain tumor subregions, especially for postoperative complex cases.

[0019] High performance and efficiency are considered: through the parameter-efficient fine-tuning strategy, the present application fully utilizes the powerful capabilities of SAM while greatly reducing the training cost and overfitting risk.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, incorporated into the specification and constituting a part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0022] Figure 1 is a flowchart of a self-prompt multi-modal malignant brain tumor segmentation method based on SAM and parameter-efficient fine-tuning according to an exemplary embodiment.

[0023] Figure 2 is a schematic diagram of the overall structure of the segmentation model according to an example embodiment, in which the top is a high-level process overview, and the three areas below are detailed structure diagrams of key modules such as multi-modal feature adaptive fusion, lightweight prompt encoder, and medical heuristic attention mechanism.

[0024] Figure 3 is a visualization result diagram of the dense prompts automatically generated by the lightweight prompt encoder for different tumor sub-regions according to an example embodiment.

[0025] Figure 4 is a segmentation effect comparison diagram of the present method and existing methods according to an example embodiment, in which (a) is the real label, (b) is the segmentation result of the present method, (c) is the segmentation result of nnU-Net, and (d) is the segmentation result of 3D U-Net.

[0026] Figure 5 is a segmentation error analysis diagram of the present method and existing methods according to an example embodiment, in which (a) is the original MRI, (b) is the error analysis of the present method, (c) is the error analysis of nnU-Net, and (d) is the error analysis of 3D U-Net.

[0027] Figure 6 is a block diagram of a self-prompting multi-modal malignant brain tumor segmentation device based on SAM and parameter efficient fine-tuning according to an example embodiment.

[0028] Figure 7 is a schematic diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0029] The example embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to designate the same elements, unless otherwise indicated. The implementations described in the following example embodiments are not meant to represent all implementations consistent with the present application.

[0030] The terminology used in the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the present application and the accompanying claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0031] It should be understood that, although the terms first, second, third, etc. can be employed in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of the present application, first information can also be referred to as second information, and similarly, second information can also be referred to as first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0032] Figure 1 is a flowchart of a self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning according to an exemplary embodiment, as shown in Figure 1 , the method can include the following steps: S1: obtaining multi-modal medical image data of a target region and pre-processing; Specifically, in a preferred embodiment, the obtained raw data contains medical image data of at least two modalities, wherein at least one of the at least two modalities is a reference modality providing anatomical structures of the target region; the medical image data includes but is not limited to magnetic resonance imaging (Magnetic Resonance Imaging, MRI), computed tomography (Computed Tomography, CT) or positron emission tomography (Positron Emission Tomography, PET) images. After obtaining the raw data, the data is pre-processed, including but not limited to data enhancement and Z-score standardization, etc. The pre-processed multi-modal medical image data is input A.

[0033] S2: based on the pre-processed multi-modal medical image data, a segmentation mask of the target region is obtained by using a self-prompting multi-modal malignant brain tumor segmentation model, the self-prompting multi-modal malignant brain tumor segmentation model includes a multi-modal feature adaptive fusion module, a lightweight prompt encoder, a SAM and a class converter, the multi-modal feature adaptive fusion module generates a corresponding deep image embedding based on the pre-processed multi-modal medical image data, the lightweight prompt encoder generates a prompt embedding for different tumor sub-regions based on the deep image embedding; based on the deep image embedding and the prompt embedding, the mask decoder of the SAM and the class converter generate a segmentation mask; (1) based on the pre-processed multi-modal medical image data, a deep image embedding is generated by using a multi-modal feature adaptive fusion module (Multi-modal Feature Adaptive Fusion Module, MFAM); The multimodal feature adaptive fusion module is used to efficiently fuse multimodal information. After the preprocessed multimodal medical image input A is sent to this module, it first performs modal grouping: Each input modality is mapped to two groups according to predetermined rules or adaptive soft assignment (e.g., when including T1, T1Gd / T1c, T2, and FLAIR, it can be mapped to "anatomical related group / fluid-edema related group," etc.). Each group undergoes preliminary feature extraction at the group level via a parallel pathway. Subsequently, the two group-level features are recalibrated by the input adapter and... The projection is a 3-channel view, fed into a pre-trained SAM image encoder, to obtain two sets of early embeddings. In the decoupling and recombination module, two types of components are first obtained: ① Shared components , representing the common structure of two sets of features; ② Group of difference components This indicates the complementary differences between the two groups.

[0034] Will passing through lightweight nonlinear elements respectively (All can be from) (Convolution + activation implementation) Then splice them together in the channel and pass through Convolution + GroupNorm + ReLU recombination into enhanced representations .

[0035] Subsequently, a medical heuristic attention mechanism (TransMedAttention) and a direct projection branch are used to obtain the depth image embedding. Specifically, the medical heuristic attention mechanism decomposes R into lesion / boundary / context sub-components, and obtains weighted features after gating and multi-scale interaction. To suppress spurious correlations and preserve direct evidence, where direct evidence refers to low-level cues such as intensity / edges from the original image, like high signal in the enhancement area or high signal in FLAIR edema, the direct projection branch will transfer the preprocessed multimodal medical image data... by Convolution → Normalization → ReLU → Convolution → Normalization → ReLU mapping to a direct feature P with the same dimension as R, with adaptive pooling used to align the spatial dimensions with R if necessary. Then, spatial gating fusion is performed: first, the weighted features... Normalize the through-feature P and concatenate them along the channel dimension. Then, feed the concatenated features into the pixel-level gate Z to obtain the depth image embedding. The pixel-level gating Z includes a [structure / mechanism / etc.] convolution, depthwise separable convolution and convolution constitute a lightweight bottleneck network, and realize gated fusion through a sigmoid activation function.

[0036] Specifically, the medical heuristic attention mechanism is used to weight the enhanced representation, as Figure 2 shown, by projecting the input enhanced representation into three semantic subspaces corresponding to lesions, boundaries, and contexts respectively, and further interacting and enhancing the features of the subspaces through a multi-path gated modulation mechanism; specifically, the first gate generated by the lesion features weights the boundary features, while the second gate generated by the context features weights the lesion features, so as to realize fine representation of tumor heterogeneity.

[0037] Specifically, the input feature dimension of the medical heuristic attention (TransMedAttention) is , which is consistent with the number of channels output by the SAM image encoder. This module includes three parts: three subspace projection, bidirectional gated modulation, residual fusion and output.

[0038] 1) Three subspace projection (lesion / boundary / context) uses three groups of 1x1 convolution (or equivalent linear mapping) to project the enhanced representation into three semantic subspaces; each projection can be followed by normalization and activation. Let the number of channels of each subspace be (default ). Formula:

[0039] wherein: is the decoupled and reorganized enhanced representation, with a size of (this embodiment C'=256). is the convolution (or linear mapping) of , l, b, and c correspond to lesions, boundaries, and contexts respectively; each mapping can be followed by GroupNorm / LayerNorm and ReLU / GELU. is the three-way subspace feature, with a size of , which enters the subsequent gated and interactive enhancement.

[0040] 2) Bidirectional gated modulation (lesion→boundary; context→lesion) Each gate is composed of a channel gate and a spatial gate, and the final gate is the product of the channel gate and the spatial gate; if only one of them is enabled, the other gate takes all 1. ​

[0041] (i) Lesion -> Boundary: only enable spatial gate, channel gate is all 1 Spatial gate , size . Where: is a 3x3 convolution; is a normalization (e.g. GroupNorm); is a function that compresses to [0,1]; is the lesion subspace feature.

[0042] Synthesis gating and modulation: , denotes broadcasting multiplication along dimensions; Thus, the boundary feature , denotes element-wise multiplication.

[0043] (ii) Context -> Lesion: only enable channel gate, spatial gate is all 1 Channel gate , size . Where: is a global average pooling; is an activation function ReLU / GELU; will channel , will , r is the SE compression ratio (e.g. 8); is the context subspace feature.

[0044] Synthesis gating and modulation: ; Thus, the lesion feature .

[0045] 3) Multiplicative attention fusion and output The lesion feature after bidirectional gating modulation and the boundary feature are fused (e.g. by weighted summation) to obtain a preliminary enhanced feature; the enhanced feature is further processed by a multi-scale module (e.g. using a dilated convolution) and a final feature enhancement module to generate a final attention map Graph; the attention map Graph is element-wise multiplied (Hadamard product) with the enhanced representation to obtain a weighted output feature Y: The present scheme is applicable to input modalities and missing modality scenarios. For M modalities X1, X2, …, X MThe learnable soft-assignment weights are used to map them to two groups; if there are missing modes, they are masked during the routing and aggregation stages, and the overall process remains unchanged.

[0046] (2) Based on the depth image embedding, a Lightweight Prompt Encoder (LPE) is used to generate cue embeddings for different tumor subregions; Specifically, the lightweight prompt encoder, by fusing prior information on anatomical location, can replace manual interaction and automatically generate dense prompts to guide segmentation, such as... Figure 2 As shown, the input is the depth image embedding output by MFAM. The data processing procedure of the lightweight prompt encoder includes: (i) The depth image embedding B simultaneously enters the local detail path and the global context path to perform feature extraction in parallel. The local detail path uses depth-separable convolution to capture the fine texture and boundary contour of the target region, while the global context path uses dilated convolution to obtain a wider receptive field to understand the spatial topological relationship between the target region and the surrounding anatomical structures. Local detail pathways (emphasizing edges / texture): in, Embedding for depth images; Depthwise convolution, Pointwise Convolution, the two constitute depthwise separable convolution; For activation functions (such as GELU / RELU). GroupNorm (group normalization); The number of output channels (which can be taken from the formula) or The dimensionality reduction setting; in one embodiment, it is fixed as .

[0047] Global contextual pathway (expanding the receptive field): in, For the set of void ratios (e.g.) Each branch should be used first. Unify the channels to the same dimension, then perform parallel dilated convolution and sum element by element. , Same as above.

[0048] (ii) The outputs of the two branches are dynamically weighted and merged by a spatial gated fusion module (SpatialGatedFusion); Specifically, the local detail branch features and the global context branch features are concatenated along the channel dimension (after normalization).

[0049] The concatenated features are fed into a gating network composed of 1x1 convolution and Sigmoid activation function, generating a spatial gating weight map .

[0050] The final fused features are obtained by weighted merging of the two branches: 3) Efficient anatomical position encoding (EAPE) The efficient anatomical position encoding module performs spatial context enhancement, in which it adaptively fuses three kinds of position priors: topological prior from learnable anatomical template, absolute coordinate prior from sinusoidal encoding, and geometric centrality prior from radial distance encoding, providing content-aware spatial context for the subsequent cue generation; Specifically, the fused features are injected with geometric / anatomical priors. This module contains three parallel position encoding components: Anatomical position encoding component: obtained by upsampling a learnable low-resolution anatomical template, denoted as ; Sinusoidal coordinate position encoding component: generated by learnable frequency and weight, denoted as ; Radial distance position encoding component: encoded by a small MLP on the distance to the image center, denoted as .

[0051] Crucially, this module contains an adaptive weight controller, which consists of adaptive average pooling, convolution, and Softmax. This controller receives the fused features as input and dynamically generates three weights satisfying These three weights are used to dynamically weight the above three position encoding components, i.e. .

[0052] The final position encoding P is the result of the fusion of the three dynamically weighted encoding components. The position encoding is injected into the fused features: 4) The enhanced features are decoded by the multi-class prompt projector into dense prompts Q for different tumor sub-regions. Each sub-region prompt map in the dense prompts Q is directly sent to the prompt encoder of SAM for encoding, forming the final prompt embedding C.

[0053] Figure 3 The visualization of the dense prompts, which are encoded by the prompt encoder of SAM, forms the final prompt embedding C.

[0054] (3) Based on the depth image embedding and the prompt embedding, the segmentation mask is generated by using the mask decoder and the class converter of SAM; Specifically, the prompt embedding C generated by LPE and the depth image embedding B generated by MFAM jointly act on the decoding stage. As shown in Figure 2 , the two-way information finally converges at the mask decoder of SAM, and the mask decoder generates the final segmentation result by using the two-way information. Then, the segmentation result passes through a trainable class converter and is mapped into the final multi-class segmentation mask D.

[0055] The class converter is specifically: Mask decoder output pixel-level features . The class converter is first refined by a small number of convolution + normalization + activation, and then mapped to K classes by one layer convolution. Then, the multi-class mask D is generated by channel dimension Softmax and pixel-wise Argmax. The calculation is summarized as: Wherein represents a light refinement sequence composed of several convolution blocks (convolution→normalization→activation).

[0056] In practice, a hybrid parameter-efficient fine-tuning (PEFT) strategy is employed to train the self-provoking multimodal malignant brain tumor segmentation model. During the training phase, the backbone parameters of the SAM's image encoder, cue encoder, and mask decoder remain frozen; only the Multimodal Feature Adaptive Fusion (MFAM), Lightweight Cue Encoder (LPE), category converter, and lightweight adapters inserted into several layers of the SAM image encoder and mask decoder are updated. During the inference phase, all weights are only computed forward without parameter updates. In practice, the number of trainable parameters accounts for less than 10% of the total parameters, effectively preserving the powerful prior knowledge of SAM while significantly reducing the risk of overfitting and computational overhead.

[0057] To effectively address the challenges of class imbalance and blurred boundaries commonly found in medical image segmentation, the training process of this invention preferably uses DiceFocalLoss as the loss function, which is the Dice loss. and Focal loss The weighted combination is defined as: in, , Dice loss is calculated using the corresponding loss weights. The formula used to optimize region overlap is: in, For the model to pixels The predicted probability, Let N be the corresponding real label, and N be the total number of pixels. To prevent smoothing coefficients with denominators of 0, Focal loss is used. This is used to focus the model on samples that are difficult to classify. The formula is: in, This represents the model's predicted probability for the correct class. To focus parameters, These are the category weight parameters.

[0058] In one embodiment of the invention, segmentation robustness is improved by dynamically adjusting the focusing parameters and the category weights for different tumor subregions (e.g., necrotic / non-enhancing tumor core (NETC), surrounding edema region (SNFH), enhancing tumor (ET), and resection cavity (RC)) to adapt to the unique challenges of preoperative and postoperative datasets respectively.

[0059] Embodiments of the present application are implemented in the following environment: the operating system is Linux, the programming language is Python 3.9, and the deep learning framework is PyTorch 2.5.0. All experiments are performed on a server equipped with an NVIDIA 4090D graphics processing unit (GPU), and the training is performed using a distributed data parallel (DDP) strategy.

[0060] In an embodiment, the disclosed BraTS 2025 challenge dataset is used for experimental verification. This dataset uses multi-modal MRI scans (T1n, T1c, T2w, T2f) containing 250 preoperative and 200 postoperative cases, which can comprehensively evaluate the performance of the present method in different clinical scenarios. In the data preprocessing stage, all images are standardized by Z-score, and data augmentation strategies including random flipping and rotation are applied to improve the robustness of the model.

[0061] Referring to Figure 4 and Figure 5 To verify the effectiveness of the present method, two mainstream segmentation networks, 3D U-Net and nnU-Net, are selected as benchmark models for comparison. As shown in the figure, whether for preoperative or postoperative complex cases, the segmentation results of the present method (Ours Prediction) are closer to the true label (Ground Truth) in terms of tumor sub-region outline and details. Error analysis also shows that the misclassification area of the present method is significantly smaller. Quantitative results also prove the superiority of the present application. On the preoperative and postoperative datasets, the average Dice of the present method is 0.9151 and 0.8820, respectively, which is significantly better than the comparison methods.

[0062] Corresponding to the foregoing embodiment of the self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning, the present application also provides an embodiment of a self-prompting multi-modal malignant brain tumor segmentation device based on SAM and parameter efficient fine-tuning.

[0063] Figure 6 is a block diagram of a self-prompting multi-modal malignant brain tumor segmentation device based on SAM and parameter efficient fine-tuning according to an exemplary embodiment. Referring to Figure 6 , the device can include: a data acquisition and preprocessing unit 21 for acquiring multi-modal medical image data of a target region and performing preprocessing; The multi-modal segmentation unit 22 is configured to obtain a segmentation mask of a target region by using a self-prompting multi-modal malignant brain tumor segmentation model based on the preprocessed multi-modal medical image data, the self-prompting multi-modal malignant brain tumor segmentation model comprising a multi-modal feature self-adaptive fusion module, a lightweight prompt encoder, a SAM, and a class converter, the multi-modal feature self-adaptive fusion module being configured to generate a corresponding deep image embedding based on the preprocessed multi-modal medical image data, the lightweight prompt encoder being configured to generate a prompt embedding for different tumor sub-regions based on the deep image embedding; and the mask decoder of the SAM and the class converter being configured to generate a multi-class segmentation mask based on the deep image embedding and the prompt embedding.

[0064] As to the apparatus in the above-described embodiments, the specific manner in which the various modules perform operations has been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0065] As to the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts are described in the part of the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purposes of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0066] Correspondingly, the present application also provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning as described above.

[0067] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning as described above. As Figure 7 As shown in the figure, a hardware structure diagram of the self-prompting multi-modal malignant brain tumor segmentation system based on SAM and parameter efficient fine-tuning provided by the embodiment of the present application is in any device with data processing capability. In addition to the Figure 7 As shown in the figure, the processor, the memory, and the network interface, any device with data processing capability in which the apparatus in the embodiment is usually based on the actual function of the device with data processing capability. In addition, it can also include other hardware, which will not be described here.

[0068] Correspondingly, the application also provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to realize the self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning as described above. The computer readable storage medium can be an internal storage unit of any device with data processing capability, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any device with data processing capability and the external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the device with data processing capability, and can also be used to temporarily store data that has been output or will be output.

[0069] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application embrace any and all variations of the application that fall within the scope of the general inventive concept as defined in the claims and that the application include all modifications, equivalents and alternatives falling within the scope of the application.

Claims

1. A self-prompting multi-modal malignant brain tumor segmentation method based on SAM and parameter efficient fine-tuning, characterized in that, The method comprises: acquiring and preprocessing multi-modal medical image data of a target region; based on the preprocessed multi-modal medical image data, using a self-prompting multi-modal malignant brain tumor segmentation model to obtain a segmentation mask of the target region, the self-prompting multi-modal malignant brain tumor segmentation model comprising a multi-modal feature adaptive fusion module, a lightweight prompt encoder, a SAM, and a class converter, the multi-modal feature adaptive fusion module generating corresponding deep image embeddings based on the preprocessed multi-modal medical image data, the lightweight prompt encoder generating prompt embeddings for different tumor sub-regions based on the deep image embeddings; based on the deep image embeddings and the prompt embeddings, the mask decoder of the SAM and the class converter generate a multi-class segmentation mask.

2. The method of claim 1, wherein, In the multi-modal medical image data, at least one is a reference modality providing the anatomical structure of the target region.

3. The method of claim 1, wherein, In the multi-modal feature adaptive fusion module: The preprocessed multi-modal medical image data is mapped to two groups according to a predetermined rule or adaptive soft allocation; Each group performs primary feature extraction through a parallel channel and inputs the early embedding corresponding to each group of primary features into a pre-trained SAM image encoder through an input adapter; Based on the early embeddings of each group, calculate the shared component and the group difference component, and reorganize them across channels into enhanced representations; Based on the enhanced representations, refine the weighted features through a medical heuristic attention mechanism, process the preprocessed multi-modal medical image data through a straight-through projection branch to obtain straight-through features, and fuse the weighted features and the straight-through features through a pixel-level gate to obtain deep image embeddings.

4. The method of claim 3, wherein, The medical heuristic attention mechanism is used to weight the enhanced representations by projecting the input enhanced representations into three semantic subspaces corresponding to lesions, boundaries, and contexts, and further interacting and enhancing the features of the subspaces through a multi-path gating modulation mechanism, specifically: Project the original multi-modal input into the lesion subspace, the boundary subspace, and the context subspace; Modulate the boundary subspace features using the lesion subspace features through the gate composed of the channel gate and the spatial gate to obtain the boundary features, and modulate the lesion subspace features using the context subspace features to obtain the lesion features; Fuse the boundary features and the lesion features, and generate an attention map through a multi-scale module and a feature enhancement module, multiply the attention map with the enhanced representations element by element to obtain the weighted features.

5. The method of claim 1, wherein, The data processing process of the lightweight prompt encoder includes: The depth image embedding enters a local detail path and a global context path in parallel for feature extraction, wherein the local detail path is used for extracting local detail features , and the global context path adopts a global feature extraction method , wherein is a depth image embedding, , , Pointwise convolution, Depthwise convolution, convolution, is group normalization, is an activation function, is the number of output channels, is a set of hole rates; Dynamically weight and merge the local detail features and the global features through a spatial gating fusion module to obtain fused features; Enhance the spatial context of the fused features through an efficient anatomical position encoding module, and decode the enhanced features into dense prompts for different tumor sub-regions through a multi-class prompt projector, and encode the dense prompts through the prompt encoder of the SAM to form prompt embeddings.

6. The method of claim 5, wherein, In the efficient anatomical position encoding module: On fusion features injecting geometry / anatomy priors, generating three parallel position encoding components, respectively anatomical position encoding component , sinus coordinate position encoding component , radial distance position encoding component ; by an adaptive weight controller, based on the fused features , dynamically generating position encoding components corresponding three weights , and ; weighting the position encoding components with the weights, resulting in a position encoding ; encoding the position injecting fusion features , resulting in enhanced features .

7. The method of claim 1, wherein, based on the depth image embedding and the prompt embedding, a mask decoder of the SAM and the class converter generate a multi-class segmentation mask, comprising: based on the depth image embedding and the prompt embedding, a segmentation result is generated by using a mask decoder; based on the segmentation result, a multi-class segmentation mask is generated by the class converter; Among them, in the category converter, the pixel-level features output by the mask decoder are refined by the combination of convolution, normalization and activation, and mapped to a plurality of category classes by one layer of convolution, and a multi-category mask is generated by channel dimension Softmax and pixel-by-pixel Argmax. , by convolution, normalization, activation combination, to a layer convolution mapping to a plurality of category classes, and a multi-category mask is generated by channel dimension Softmax and pixel-by-pixel Argmax.

8. The method of claim 1, wherein, The multi-modal feature adaptive fusion module, the lightweight prompt encoder, the class converter and the SAM are all trained network structures, wherein the main parameters of the SAM are kept frozen during training, and a plurality of lightweight adapter parameters are inserted; In the training process, DiceFocalLoss is used as the loss function, which is defined as: ， wherein, , is the corresponding loss weight, and the Dice loss is used to optimize the region overlap, and its formula is: ， wherein is the prediction probability of a pixel by the model, is its corresponding true label, and N is the total number of pixels, is a smoothing coefficient to prevent the denominator from being 0; Focal loss is used to focus the model on difficult-to-classify samples, and its formula is: ， wherein, is the predicted probability of the model for the correct class, is the focus parameter, is the class weight parameter.

9. An electronic device, comprising: comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7.

10. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed by the processor to implement the steps of the method of any one of claims 1-7.

Images