Medical image segmentation method based on inter-layer feature fusion and balanced expert hints

Abstract

This application discloses a medical image segmentation method based on inter-layer feature fusion and balanced expert input. The method acquires the medical image to be segmented and the prompt information, and inputs the medical image into an image encoder. Low-dimensional bottleneck features are extracted from at least one non-first encoding layer. Multiple convolutional expert outputs are weighted and fused based on routing weights to obtain enhanced bottleneck features. The enhanced bottleneck features from previous layers are read, spatially aligned, aggregated, channel projected, and upsampled to form historical bias features, which are then injected into the main path output. A manifold-constrained multi-branch transformation including residual mixing mapping is performed in the main path, where the residual mixing mapping matrix elements are non-negative and the sum of each row element and the sum of each column element are both 1. Image embedding features are obtained based on the enhanced output features. The prompt information is encoded to obtain prompt embedding features, and then decoded to obtain the segmentation result. This application is beneficial for improving the accuracy, stability, and applicability of medical image segmentation.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a medical image segmentation method, electronic device and storage medium based on interlayer feature fusion and balanced expert prompts. Background Technology

[0002] Medical image segmentation is a crucial and fundamental step in medical image analysis, with significant applications in various medical image processing methods, including computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, fundus imaging, and endoscopic imaging. Accurate segmentation of organs, tissues, lesions, or other target regions provides fundamental data support for clinical diagnosis, lesion assessment, surgical planning, and treatment outcome analysis. Therefore, obtaining segmentation results that balance accuracy, stability, and applicability across different medical image scenarios has become a persistent challenge in this field.

[0003] In recent years, with the development of deep learning technology, medical image segmentation methods based on convolutional neural networks, visual Transformers, and cue-based segmentation frameworks have been widely applied. In particular, cue-based medical image segmentation schemes can guide the target region through point cues, bounding box cues, or other cue information, thereby improving the flexibility of the segmentation process. However, in complex medical image scenarios, existing schemes still have the following shortcomings: On the one hand, existing segmentation models often rely on a single structure or fixed feature enhancement path, which easily leads to insufficient adaptability and unstable segmentation performance when facing different modalities, organs, and image quality conditions. On the other hand, multi-layer network structures are prone to insufficient feature reuse in previous layers during layer-by-layer propagation, resulting in insufficient utilization of shallow detail information and historical semantic information, thus limiting the model's feature representation ability. Furthermore, although some existing schemes introduce expert selection mechanisms or multi-branch enhancement mechanisms, their expert activation methods are usually relatively fixed, making it difficult to achieve more targeted enhancement processing based on current image features; simultaneously, cross-layer information may be amplified or attenuated during fusion and transmission, leading to instability in deep propagation.

[0004] Therefore, how to improve the segmentation accuracy, applicability, and operational stability of the model within the framework of prompting medical image segmentation has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a medical image segmentation method, electronic device, and storage medium based on interlayer feature fusion and balanced expert, to at least solve the technical problems of insufficient interlayer feature reuse, limited flexibility of expert enhancement, and insufficient stability of deep propagation in existing medical image segmentation schemes.

[0006] To address the aforementioned technical problems, this application provides a medical image segmentation method based on inter-layer feature fusion and balanced expert input, comprising: acquiring a medical image to be segmented and prompt information; inputting the medical image to be segmented into an image encoder; extracting low-dimensional bottleneck features from the current encoding layer features in at least one non-first encoding layer, after multi-head attention processing and before residual connections; weighting and fusing multiple convolutional expert outputs using routing weights generated based on the low-dimensional bottleneck features to obtain enhanced bottleneck features; reading the enhanced bottleneck features from the preceding layer, and obtaining historical bias features through spatial alignment, aggregation, channel projection, and upsampling, and injecting them into the current encoding layer backbone path output; performing a manifold-constrained multi-branch transformation containing at least a residual mixing map in the current encoding layer backbone path, wherein the elements in the matrix corresponding to the residual mixing map are non-negative, and the sum of the elements in each row and the sum of the elements in each column are both 1; obtaining enhanced output features based on the backbone path output after injection and manifold-constrained multi-branch transformation; obtaining image embedding features based on the enhanced output features, encoding the prompt information to obtain prompt embedding features, and decoding the image embedding features and prompt embedding features to obtain the segmentation result.

[0007] Furthermore, the step of extracting low-dimensional bottleneck features from the current coding layer features includes: performing spatial compression and channel compression on the current coding layer features to obtain bottleneck features with both spatial resolution and channel dimension lower than those of the current coding layer features.

[0008] Furthermore, the plurality of convolution experts includes at least three convolution experts with different kernel sizes.

[0009] Furthermore, the aggregation includes: performing mean aggregation on the enhanced bottleneck features of the spatially aligned preceding layers to obtain historical fusion features, and performing channel projection and upsampling based on the historical fusion features.

[0010] Furthermore, the multi-branch transformation based on manifold constraints includes: pre-mapping the multi-branch flow features of the current coding layer, obtaining the transformed features through sub-layer operations and post-mapping of the current layer, and performing residual mixing on the multi-branch flow features of the current coding layer through the residual mixing mapping.

[0011] Furthermore, the prompt information includes one or more foreground points selected from the foreground pixel region of the target area.

[0012] Further, the step of decoding the segmentation result based on the image embedding feature and the cue embedding feature includes: inputting the position encoding, the image embedding feature, and the cue embedding feature into a mask decoder to obtain a decoded marker feature and a decoded image feature; generating a dynamic mask weight based on the decoded marker feature, and generating a predicted mask based on the dynamic mask weight and the decoded image feature; and restoring the size of the predicted mask to obtain the segmentation result.

[0013] Furthermore, during the training phase, the loss between the predicted mask and the corresponding labeled mask ground truth is calculated. The loss calculation uses a composite loss function consisting of Dice loss and cross-entropy loss. During the inference phase, the predicted mask after size recovery is subjected to probability mapping and threshold binarization to obtain the segmentation result.

[0014] This application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the electronic device performs the prompting medical image segmentation method described in any of the above technical solutions.

[0015] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the processor performs the prompting medical image segmentation method described in any of the above technical solutions.

[0016] Compared with the prior art, the above technical solution has at least the following beneficial effects: By introducing inter-layer feature fusion and balanced expert collaboration mechanism into the prompting medical image segmentation framework, this application enables the model to take into account the current layer feature enhancement, previous layer information reuse and stable transmission of multi-branch information during the encoding process, thereby improving the feature expression ability and cross-layer semantic coherence in complex medical image scenarios, reducing the instability risk in the deep propagation process, and enhancing the accuracy, stability and applicability of the segmentation results under prompting guidance.

[0017] The above-mentioned technical solutions also have the following advantages: In some implementations, low-dimensional bottleneck feature extraction helps to retain key semantic information while controlling computational overhead; expert configuration with different receptive fields helps to balance local details and contextual information in modeling; alignment and aggregation of historical features helps to improve the stability of inter-layer information reuse; constrained multi-branch transformation helps to enhance the controllability of information mixing process; the method of using foreground points to form prompts helps to improve the clarity of target guidance and the flexibility of segmentation processing; the method of generating dynamic masks based on decoded features helps to improve the adaptability of segmentation output to different targets and different scenarios; the training phase uses a composite loss function consisting of Dice loss and cross-entropy loss for supervision, and the inference phase performs probability mapping and threshold binarization on the prediction mask, thereby helping to balance the model training effect and the final output quality. Attached Figure Description

[0018] The present application will be further described below with reference to the accompanying drawings and embodiments: Figure 1 A flowchart illustrating the overall process of a medical image segmentation method based on interlayer feature fusion and balanced expert suggestions provided in this application; Figure 2 A schematic diagram of the medical image segmentation framework based on interlayer feature fusion and balanced expert suggestions provided in this application; Figure 3 This is a schematic diagram of BMOE feature fusion provided in this application; Figure 4 A flowchart illustrating step S3 provided for this application; Figure 5 A schematic diagram of the inter-layer historical feature fusion and reuse process provided for this application; Figure 6 A flowchart illustrating step S4 provided for this application; Figure 7 A flowchart illustrating step S5 provided for this application; Figure 8 The image shows the results of the visual cup and visual disk segmentation in this application on the REFUGE2 dataset; Figure 9 The image shows the melanoma segmentation results of this application on the ISIC2016 dataset; Figure 10 This is a comparison of ablation experiments performed on the ISIC2016, REFUGE2, and TMNIX datasets. Detailed Implementation

[0019] The following detailed description of the content of this application is provided in conjunction with specific embodiments. It should be noted that in this application, the terms "comprising," "including," and their variations are used to indicate the existence of the listed elements or steps, but do not exclude the existence of other elements, steps, or combinations thereof; "at least one" indicates one or more; "multiple" indicates two or more; "and / or" indicates any one, any several, or all of the related objects; expressions such as "first," "second," and "third" are only used to distinguish different objects, steps, or modules, and are not used to limit the order, quantity, priority, or importance. Unless otherwise explicitly stated, expressions such as "input," "output," "generate," "obtain," "extract," "fusion," "inject," "map," and "restore" can all be understood as data processing, feature processing, or result processing processes implemented through software, hardware, or a combination of software and hardware; expressions such as "feature," "feature map," "embedded feature," and "output feature" can all be understood as intermediate representations or result representations obtained after data has undergone corresponding processing; expressions such as "module," "unit," "network," "encoder," and "decoder" can refer to software programs, hardware circuits, logical structures, or combinations thereof with corresponding functions. Each step, module, or processing procedure described in this application may be executed in the order specified in the specification without departing from the technical concept of this application, or may be adapted to the actual application scenario.

[0020] In this application, medical image segmentation generally refers to the processing task of dividing organs, tissues, lesions or other target areas in medical images into pixel-level or region-level segments; terms such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Vision Transformer (ViT), Prompt Encoder, and Mask Decoder are used in their usual meaning in this art. Segment Anything Model (SAM) typically refers to a general segmentation model that can perform segmentation based on prompts. Medical Segment Anything Model (MedSAM) typically refers to a SAM-like model adapted or fine-tuned for medical imaging scenarios. Mixture of Experts (MoE) typically refers to a feature processing structure composed of multiple expert branches and routing mechanisms. Balanced Mixture of Experts (BMOE) typically refers to introducing a more balanced and constrained mixing method in the process of expert selection or expert fusion. Inter-layer History Feature Manifold Fusion (IHFMF) typically refers to a processing mechanism that fuses and reuses historical features between different layers and combines them with constrained feature propagation methods to enhance the stability of information transmission. A token typically refers to a vector unit in a Transformer structure used to represent an image patch, cue information, or decoding token; a patch typically refers to a local image patch obtained by dividing the input image; and logits typically refer to the raw output score before probability mapping. Terms such as softmax, sigmoid, Dice loss, cross-entropy loss, DiceCELoss, Intersection over Union (IoU), depthwise separable convolution, 2D layer normalization, upsampling, channel projection, and positional encoding are used as commonly understood in the art. REFUGE2, ISIC2016, and TNMIX represent commonly used medical image segmentation datasets in the art, used for training, validating, or testing models or methods. In this application, unless otherwise specified, the above terms, model names, dataset names, and English abbreviations have their commonly understood meanings.

[0021] Medical image segmentation is a crucial and fundamental step in medical image analysis, with significant applications in computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, fundus imaging, and endoscopic imaging. While existing medical image segmentation methods widely employ convolutional neural networks, visual Transformers, and cue-based segmentation frameworks, they still generally suffer from insufficient model versatility, inadequate reuse of historical features across layers, and relatively fixed expert selection mechanisms in complex medical image scenarios. These issues can easily lead to unstable segmentation performance, limited feature representation capabilities, and the risk of gradient vanishing or exploding during deep propagation.

[0022] To address this, this application proposes a cue-based medical image segmentation method based on inter-layer feature fusion and balanced expert input. Within the framework of cue-based medical image segmentation, this method combines low-dimensional bottleneck feature enhancement, reuse of historical features from preceding layers, and a constrained multi-branch stable propagation mechanism to achieve collaborative processing of image encoding enhancement, cue-guided segmentation, and segmentation result output, thereby improving the accuracy, stability, and applicability of medical image segmentation.

[0023] Figure 1 This application provides an overall flowchart of a medical image segmentation method based on inter-layer feature fusion and balanced expert suggestions. Figure 1As shown, the medical image segmentation method based on inter-layer feature fusion and balanced expert prompts provided in this application includes at least the following steps: In step S1, the medical image to be segmented and prompt information are obtained, and the medical image to be segmented is preprocessed so that the preprocessed medical image to be segmented meets the requirements of subsequent image coding processing; In step S2, low-dimensional bottleneck features are extracted from at least one non-first coding layer, and multiple convolutional expert outputs are weighted and fused based on routing weights to obtain enhanced bottleneck features, so as to realize expert enhancement processing for the current image features; In step S3, the enhanced bottleneck features of the previous layer are read and historical bias features are formed, and injected into the main path output of the current coding layer to realize the fusion and reuse of inter-layer historical features; In step S4, manifold constraint multi-branch transformation is performed in the main path of the current coding layer to enhance the stability of the multi-branch information transmission process; In step S5, image embedding features are obtained based on enhanced output features, the prompt information is encoded, and the segmentation result is decoded to complete the prompt segmentation processing of the medical image to be segmented. In this embodiment, the routing weights in step S2 can be generated from low-dimensional bottleneck features and used to perform weighted fusion of the outputs generated by multiple convolutional experts based on low-dimensional bottleneck features; the historical bias features in step S3 can be formed by spatial alignment, aggregation, channel projection, and upsampling of the enhanced bottleneck features of the preceding layer; the manifold-constrained multi-branch transformation in step S4 can at least include residual mixing mapping, and the elements in the matrix corresponding to the residual mixing mapping are non-negative, and the sum of the elements in each row and the sum of the elements in each column are both 1; the process of obtaining the segmentation result in step S5 can further include: obtaining image embedding features based on enhanced output features, encoding the prompt information to obtain prompt embedding features, and decoding based on the image embedding features and prompt embedding features to obtain the segmentation result.

[0024] Figure 2 This is a schematic diagram of the medical image segmentation framework based on inter-layer feature fusion and balanced expert suggestions provided in this application. Figure 2 As shown, the cue-based medical image segmentation framework of this application includes at least an input image branch, a cue information branch, a position encoding branch, an image encoder, a cue encoder, and a mask decoder. The input image branch is used to receive the medical image to be segmented, which contains cue points. The image encoder is used to extract features from the input medical image and introduces BMOE and IHFMF into the coding block processing to achieve low-dimensional bottleneck feature enhancement, inter-layer historical feature fusion and reuse, and stable propagation along the main path, thereby outputting image embedding features. The prompt information branch is used to provide prompt information P corresponding to the target area. In this embodiment, the prompt information is a dot prompt. The prompt encoder is used to encode the prompt information P and output the prompt embedding feature. The position encoding branch is used to provide position encoding. Mask decoders are used for image embedding features. Hints about embedded features and position encoding Generate a predicted segmentation map. Therefore... Figure 2 This visually illustrates the overall collaborative relationship between image encoding, cue encoding, position encoding, and mask decoding in this application.

[0025] The following is combined Figure 1 and Figure 2 Further explanation is given for steps S1 to S5.

[0026] In step S1, the medical image to be segmented and the prompt information are acquired, and the medical image to be segmented is preprocessed. Specifically, the input image is denoted as... ,in, Indicates the number of channels. and These represent the height and width of the input image, respectively; the preset channel mean vector and channel standard deviation vector are represented as follows: and After performing channel-wise normalization on the input image, a normalized image tensor is obtained. Its calculation method can be expressed as:

[0027] Through the above preprocessing, the medical image to be segmented can meet the input requirements of the subsequent image encoder, providing a unified data input basis for subsequent low-dimensional bottleneck feature extraction, balanced hybrid expert enhancement, inter-layer historical feature fusion, and prompt-guided segmentation.

[0028] In step S2, low-dimensional bottleneck features are extracted from at least one non-first encoding layer, and the outputs of multiple convolutional experts are weighted and fused based on routing weights to obtain enhanced bottleneck features. Specifically, the normalized image tensor obtained in step S1 is... After being input into the image encoder, the image is spatially divided into multiple non-overlapping segments with side lengths of [missing information]. Image patches are mapped to initial token features via a block embedding module. ,in represent height, represent The width.

[0029] Subsequently, the pre-trained position encoding Interpolation alignment is performed and the results are added to the block embedding features to obtain the initial encoded features. It can be represented as:

[0030] in, This indicates that the position will be encoded. Interpolate to size Subsequently, the initial encoded features will be... The input is processed through a multi-layer encoder for layer-by-layer feature enhancement.

[0031] In this embodiment, the first The input of the layer encoder is denoted as The output after multi-head attention is denoted as The output after semantic enhancement is denoted as Each encoder layer stores a low-dimensional bottleneck feature. ,in, These represent the height, width, and channel dimensions of the bottleneck feature, respectively. Then, a balanced hybrid expert-selective augmentation is performed on the bottleneck feature. For any spatial location... The router outputs the first The weights of each expert are:

[0032] in, This represents the total number of experts. Indicates position First The route logits corresponding to each expert This represents the temperature parameter. Then, multiple expert outputs are weighted and fused to obtain the enhanced bottleneck characteristics. It can be represented as:

[0033] in, Indicates the first Each convolutional expert uses a different kernel size to correspond to a different receptive field.

[0034] Through the above step S2, conditional expert enhancement based on current image features can be achieved in the low-dimensional bottleneck subspace.

[0035] Figure 3 This is a schematic diagram of BMOE feature fusion provided in this application. Figure 3 As shown, the input features are processed by multiple convolutional expert branches. In this embodiment, the multiple convolutional experts include at least 1×1, 3×3, and 5×5 convolutional experts, where different kernel sizes correspond to different receptive fields, used to extract structural and semantic information at different scales. Simultaneously, the input features are also used to generate expert routing weights, which characterize the contribution of each convolutional expert under the current input conditions. Then, based on the expert routing weights, the outputs of the multiple convolutional experts are gating and fused to obtain the feature output. Thus, Figure 2 This intuitively illustrates the core processing logic of BMOE feature fusion in this application, namely: instead of using a fixed activation method for multiple expert branches, it dynamically weights and fuses the outputs of different experts based on the specific representation of the input features through expert routing weights, in order to achieve more flexible conditional feature enhancement.

[0036] Furthermore, Figure 3 The BMOE feature fusion mechanism shown can be used to enhance low-dimensional bottleneck features. Specifically, in at least one non-first encoding layer, low-dimensional bottleneck features can be extracted from the features of the current encoding layer. Then, routing weights are generated based on these low-dimensional bottleneck features, and the outputs generated by multiple convolutional experts based on the low-dimensional bottleneck features are weighted and fused according to the routing weights to obtain enhanced bottleneck features. This approach enables the model to adaptively call different receptive field expert branches in the low-dimensional bottleneck subspace, improving the comprehensive modeling ability for local details, contextual information, and channel remapping information while maintaining computational complexity. This provides a more effective feature foundation for subsequent inter-layer historical feature fusion and backbone path enhancement.

[0037] Figure 4 A flowchart illustrating step S3 provided in this application. Figure 4 As shown, in step S3, the enhanced bottleneck features of the preceding layer are read and historical bias features are formed, which are then injected into the output of the current coding layer backbone path. Specifically, in step S31, the enhanced bottleneck features of the preceding layer are read; in step S32, the enhanced bottleneck features of the preceding layer are spatially aligned and aggregated to obtain historical fusion features; in step S33, the historical fusion features are channel-projected and upsampled to obtain historical bias features, which are then injected into the output of the current coding layer backbone path.

[0038] In some implementations, for the current number Layer, from the nearest The enhancement bottleneck features are read from each preceding layer, their spatial dimensions are aligned, and the average value is calculated to obtain the historical fusion features. It can be represented as:

[0039] in, Indicates the first The number of historical layers read by the layer. This indicates an operation that aligns historical bottleneck features to a uniform spatial size. Historical bias features are obtained after channel projection and upsampling. It can be represented as:

[0040] in, This represents the mapping from the bottleneck channel dimension to the current backbone channel dimension. This indicates upsampling. Through step S3, the low-dimensional spatial summary formed in the previous coding layer can be introduced into the current coding layer processing, thereby realizing the fusion and reuse of historical features between layers.

[0041] Figure 5 A schematic diagram illustrating the process of fusing and reusing historical features between layers provided in this application. For example... Figure 5 As shown, after the input features enter the multi-layer coding blocks, each coding block, based on the feature extraction of the current layer, further generates corresponding low-dimensional bottleneck features, and enhances these low-dimensional bottleneck features through the BMOE module. As the coding process progresses layer by layer, the enhanced bottleneck features generated by each layer not only participate in the feature enhancement of the current layer, but can also be read and reused by subsequent coding layers. Figure 5 The illustrated process demonstrates the transmission and reuse relationship of the "enhanced bottleneck features of the preceding layer" in the multi-layer coding structure of this application. Specifically, the enhanced bottleneck features output by the preceding coding block can serve as the basic information source for constructing historical bias features in subsequent coding blocks. This allows subsequent coding layers, when performing feature modeling at the current layer, to not only utilize the input features of their own layer but also explicitly absorb low-dimensional spatial summary information from previous layers. Through this method, the fusion and reuse of historical features between layers can be achieved, enhancing the semantic coherence between different coding layers and improving the encoder's ability to utilize multi-scale structural information and historical semantic information.

[0042] In some implementations, for the current number Layers, from the most recent The enhanced bottleneck features of the preceding layers are read, spatially aligned, and aggregated to obtain historical fusion features. Then, through channel projection and upsampling, historical bias features are formed and injected into the current coding layer's backbone output. Based on the above process, Figure 5 This not only shows the hierarchical relationship between the low-dimensional bottleneck features corresponding to each coding block and the BMOE enhancement processing, but also shows the reuse path of the enhancement bottleneck features of the preceding layer in the subsequent layers. This intuitively reflects the inter-layer historical feature fusion and reuse mechanism adopted in this application, which is conducive to enhancing the semantic coherence between different coding layers, improving the utilization of effective feature information of the preceding layer, and improving the feature expression effect and result stability in the segmentation process.

[0043] Figure 6 A flowchart illustrating step S4 provided in this application. Figure 6As shown, in step S4, a manifold-constrained multi-branch transformation is performed in the current coding layer backbone path. Specifically, in step S41, the multi-branch flow features are pre-mapped in the current coding layer backbone path, and transformed features are obtained through current layer sub-layer operations and post-mapping; in step S42, the multi-branch flow features are residual-mixed through residual mixing mapping, wherein the elements of the residual mixing mapping matrix are non-negative, and the sum of the elements in each row and the sum of the elements in each column are both 1; in step S43, enhanced output features are obtained based on historical bias features and the backbone path output after manifold-constrained multi-branch transformation. In some embodiments, the single-layer propagation of the manifold-constrained multi-branch processing can be expressed as:

[0044] in, Indicates the first Multi-branch flow characteristics of the layer This indicates the multi-branch flow characteristics of the output of this layer; This indicates sub-level operations of the current layer. , , They represent the first Layer pre-mapping, post-mapping, and residual hybrid mapping. To ensure the stability of deep information propagation, residual hybrid mapping is used. Apply double random constraints to make the residual mixture mapping satisfy , This indicates the number of branches in a multi-branch flow. express A vector of all 1s. Then, the historical bias is... Injected into the main branch of the current layer, resulting in the first... The layer outputs after passing through the semantic enhancement module. It can be represented as:

[0045] in, Indicates the first The backbone output after manifold constraint processing of the layer, Indicates the first The learnable scaling factor of the layer. Through the above step S4, the features of the current layer not only include the structured transformation results of this layer, but also explicitly absorb the low-dimensional space summary of the previous several layers.

[0046] Figure 7 A flowchart illustrating step S5 provided in this application. Figure 7As shown, in step S5, image embedding features are obtained based on the enhanced output features, the prompt information is encoded, and then decoded to obtain the segmentation result. Specifically, in step S51, image embedding features are obtained based on the enhanced output features; in step S52, the prompt information is encoded to obtain prompt embedding features; in step S53, the position encoding, image embedding features, and prompt embedding features are input together into the mask decoder to obtain decoded marker features and decoded image features; in step S54, dynamic mask weights are generated based on the decoded marker features, and then a prediction mask is generated based on the dynamic mask weights and decoded image features; in step S55, the size of the prediction mask is restored to obtain the segmentation result.

[0047] In some implementations, after completing the layer-by-layer processing of all encoded blocks, the resulting batch feature dimensions are changed from... Convert to After passing through 1×1 convolution, 2D layer normalization, 3×3 convolution, and 2D layer normalization in sequence, the final output batch image embedding features are obtained. ,in, Indicates batch size.

[0048] For batch size The sample set, the first The annotation mask corresponding to each sample is denoted as . ,in, and These represent the height and width of the annotation mask, respectively. When When, it indicates the pixel position. It belongs to the truth value region of the target foreground; When, it indicates the pixel position. This belongs to the background area. Randomly selecting one or more pixels from the foreground ground truth region as prompt information yields the [number of pixels]. Foreground spots corresponding to each sample It can be represented as:

[0049] in, Indicates the first In the nth sample Two-dimensional coordinates of each foreground point. A uniform foreground label is assigned to each of the foreground points. Based on this, the coordinate information of the foreground points is packaged into a coordinate tensor. The corresponding tags are packaged into tag vectors. Input the foreground point coordinate tensor and label vector into the cue encoder to obtain the first... Sparse cue embedding features corresponding to each sample .

[0050] Furthermore, the location is encoded. Image encoding and sparse cue embedding features The common input mask decoder obtains the decoded token features. and decoding image features .right The upsampled decoded image features are obtained through the upsampling module. , , This indicates the number of feature channels after upsampling. and This represents the spatial size of the low-resolution prediction mask's logits. From... Extract the mask token output features from it, denoted as . ,in, This represents the feature dimension of the mask token.

[0051] For each mask token, the output feature is mapped through the corresponding hypernetwork to generate a corresponding dynamic mask weight, which can be represented as:

[0052] in, This represents the corresponding hypernet.

[0053] The dynamic mask weights are obtained by stacking all the dynamic weights. At the same time, the upsampled decoding features Flattened Then, the dynamic mask weights and the flattened decoded features are multiplied into a matrix to generate a low-resolution prediction mask logits, denoted as:

[0054] in, This indicates that matrix multiplication is performed along the channel dimension. This indicates that the product result is restored to a spatial grid form. Subsequently, the low-resolution prediction mask logits are interpolated to the original image input size: .

[0055] During the training phase, With the corresponding labeled mask truth value Perform loss calculation: ,in, DiceCELoss represents a composite loss function that combines Dice Loss and Cross Entropy Loss.

[0056] During the inference phase, the interpolated prediction mask logits( After Sigmoid mapping, the predicted probability map is obtained. Finally, based on the preset threshold For prediction probability map Binarization is performed to obtain the final output prediction mask:

[0057] in, This represents the coordinates of each pixel in the predicted mask. Through step S5 above, the predicted mask generation guided by prompts, size restoration, and final segmentation result output can be completed.

[0058] The following section uses the REFUGE2 dataset from the Retinal Glaucoma Challenge to further illustrate the proposed medical image segmentation method based on interlayer feature fusion and balanced expert suggestions.

[0059] The objective of this embodiment is to segment the optic disc and optic cup regions in retinal fundus images to verify the feasibility and effectiveness of this application in fundus medical image segmentation scenarios. This embodiment uses the REFUGE2 training dataset, with labels containing segmentation masks corresponding to the optic disc and optic cup. The evaluation metrics are Dice and IoU. The experimental environment is configured with an NVIDIA-RTX3090-GPU-24GB, CUDA-12.5, and the software environment is Python 3.9 and PyTorch 2.8.0. In this embodiment, the batch size is set to... =1, the input color fundus image size is set to 1024×1024, and the preset channel mean vector and channel standard deviation vector are [123.675,116.28,103.53] and [58.395,57.12,57.375], respectively. After normalizing the input image by channel, a normalized image tensor of size 1×3×1024×1024 is obtained. For the labeled mask, supervision labels are generated according to the semantic categories corresponding to different pixel values ​​in the dataset, and the background region, optic disc region and optic cup region are mapped to the corresponding category labels respectively. Furthermore, a multi-channel mask representation suitable for network training and decoding supervision is constructed to constrain the segmentation results of the optic disc and optic cup respectively.

[0060] In the image feature encoding stage, the preprocessed color fundus image is input into the image encoder. The image encoder uses the same ViT-B architecture as the MedSAM model encoder, with a patch size of 16, a token channel dimension of C=768, and the number of encoding layers L=12. The input image is spatially divided into 64×64 non-overlapping image blocks, which are then mapped to initial visual token features through a block embedding module. Subsequently, pre-trained positional encoding is added to the embedded features to obtain initial encoded features with a size of 1×64×64×768. Thus, the input image is converted into visual tokens with spatial location information, which serve as input for subsequent multi-layer encoding blocks.

[0061] Next, the initial encoded features are input into a 12-layer encoder and feature extraction is performed layer by layer. The current layer features are first updated with global semantic relationships via the backbone attention path, and then low-dimensional bottleneck features are extracted from the current layer's backbone features. These low-dimensional bottleneck features are obtained using 1 / 4 spatial compression and 1 / 4 channel compression, compressing the backbone features from 64×64×768 to 16×16×192. Afterward, non-linear semantic reorganization and inter-layer historical feature bias injection are performed. To improve the local structural representation in the low-dimensional bottleneck space, balanced hybrid expert routing enhancement is performed on the low-dimensional bottleneck features generated at each layer. The expert routing outputs pixel-level softmax weights for each spatial location, and the outputs of multiple convolutional expert branches are weighted and fused accordingly. Specifically, 3×3 depthwise separable convolutions, 5×5 depthwise separable convolutions, and 1×1 lightweight convolutions are used to extract boundary detail information, larger-range contextual information, and channel remapping information, respectively. Finally, the obtained feature dimensions are adjusted from... Convert to After passing through 1×1 convolution, 2D layer normalization, 3×3 convolution, and 2D layer normalization in sequence, while keeping the spatial resolution unchanged at 64×64, the number of channels is mapped from 768 to 256, thus obtaining the image embedding features, i.e. The size is 1×256×64×64, which will be used to generate optic disc segmentation results and optic cup segmentation results corresponding to the REFUGE2 fundus image.

[0062] In the prompt information encoding stage, for the labeled mask corresponding to the color fundus image, one or more foreground points are selected as prompt information from the target foreground pixel region. In this embodiment, three pixels are selected as prompt information. The coordinate information of the three pixels is sequentially packaged into a 3×2 tensor, and the corresponding labels are packaged into a 3×1 tensor. Then, the point coordinate tensor and label vector are input into the Prompt Encoder to generate sparse prompt embedding features Esp with a size of 1×3×256. During the training stage, point prompts are automatically constructed using the labeled mask; during the inference stage, prompts can also be automatically generated by manual interaction or coarse localization results. Unlike the Prompt Encoder of the original MedSAM model, the sparse prompt embedding features obtained in this embodiment... It will be used to generate dynamic mask weights in the subsequent decoding process to more flexibly adapt to downstream segmentation tasks.

[0063] Position encoding is performed during the decoding output and post-processing stages. Image encoding and sparse cue embedding features The common input to MaskDecoder yields the decoded token features. and decoding image features The dimensions of the two are 1×3×256 and 1×256×64×64, respectively. Upsampling Sdec yields the upsampled decoded image feature U, with a size of 1×32×256×256. This process is used to save computational resources while reducing the number of channels; from Extract the mask token output features and obtain the corresponding dynamic weights through hypernetwork mapping. Then, multiply these weights with the flattened upsampled decoded image features along the channel dimension and restore them to a spatial grid form to generate a low-resolution prediction mask logits, denoted as . The image size is 1×3×256×256. Then, the low-resolution prediction mask logits are interpolated to the original color fundus image size of 1×3×1024×1024, denoted as... During the training phase, for With the corresponding labeled mask truth value Loss calculation is performed, using DiceCELoss as the loss function; during the inference phase, After Sigmoid mapping, and then according to a preset threshold Binarization is performed using a value of 0.5 to obtain the final output prediction mask for each pixel. Thus, this embodiment implements a complete processing flow from input color fundus image to output of optic disc and optic cup region segmentation results on the REFUGE2 dataset.

[0064] To verify the effectiveness of the technical solution of this application, comparative experiments were conducted on the REFUGE2, ISIC2016 and TMNMIX datasets, and Dice and IoU were used as evaluation metrics.

[0065] Table 1 Comparison of Experimental Indicators

[0066] Table 1 presents the experimental performance comparison results of this application with models such as FAT-Net, ResUNet, Swin-Unetr, TransBTS, nnUNet, and MedSAM on multiple datasets. As shown in Table 1, this application outperforms the comparison models in segmentation of the optic disc and optic cup regions on the REFUGE2 dataset. Furthermore, this application also achieves superior Dice and IoU metrics on the ISIC2016 and TMNIX datasets. These data demonstrate that the inter-layer historical feature fusion and balanced hybrid expert collaboration mechanism proposed in this application not only improves the segmentation performance of the optic disc and optic cup in fundus images but also exhibits good generalization ability and cross-scene adaptability in different types of medical image scenarios, such as melanoma images and ultrasound images.

[0067] Specifically, on the REFUGE2 dataset, this application achieved Dice and IoU of 96.5 and 93.2 in the optic disc region, and 91.9 and 85.3 in the optic cup region, respectively; on the ISIC2016 dataset, the Dice and IoU reached 95.9 and 92.4, respectively; and on the TNMIX dataset, the Dice and IoU reached 91.3 and 84.9, respectively. Compared with MedSAM, this application shows significant improvements in all metrics on the above datasets, indicating that introducing low-dimensional bottleneck feature enhancement, fusion and reuse of previous layer historical features, and a constrained multi-branch stable propagation mechanism into the cue-based medical image segmentation framework can effectively improve image feature representation ability, historical information utilization ability, and segmentation result stability.

[0068] Table 2 Ablation Experiment Results

[0069] Table 2 presents the ablation experiment results of this application on the REFUGE2, ISIC2016, and TMNIX datasets, including a performance comparison of the MedSAM baseline model, the model without inter-layer history feature reuse (IHFMF), the model without BMOE, and the complete scheme. As shown in Table 2, after removing the inter-layer history feature reuse module or the BMOE module, the segmentation metrics on each dataset are lower than the complete scheme. This indicates that the inter-layer history feature fusion reuse mechanism and the balanced hybrid expert mechanism each contribute to the performance improvement of the model in this application, and that their combined effect further enhances the performance. Therefore, this application does not simply replace existing models locally, but rather achieves superior segmentation results within the cue-based segmentation framework through the synergistic cooperation of history feature reuse, expert enhancement, and stable propagation.

[0070] Figure 8 This image shows the segmentation results of the optic cup and optic disc on the REFUGE2 dataset, and compares them with the segmentation results of other models. Figure 8 As shown, in multiple fundus image samples, the segmentation results of this application exhibit high consistency with the ground truth region, especially when the boundaries of the optic cup and optic disc are complex or the target region is small, it still maintains good contour integrity and localization accuracy. Compared with other comparative models, this application performs better in terms of preserving local boundary details and maintaining the consistency of target region morphology, indicating that this application can effectively improve the modeling ability of structural details and semantic boundaries in fundus image segmentation tasks.

[0071] Figure 9 This image shows the melanoma segmentation results of this application on the ISIC2016 dataset, and compares them with the segmentation results of other models. Figure 9 As shown, this application can effectively maintain the shape of the target edge and the integrity of the main region in the extraction of the target region in melanoma images. Even in the presence of color changes, irregular boundaries, and background interference, it can still obtain relatively accurate segmentation contours. This further illustrates that the technical solution of this application is not only applicable to fundus image segmentation, but can also be well transferred to other medical image segmentation tasks, thus demonstrating good cross-modal and cross-scene adaptability.

[0072] Figure 10 This image shows a comparison of ablation experiments performed on the ISIC2016, REFUGE2, and TNMIX datasets based on this application. Figure 10As shown, the complete scheme outperforms the comparison schemes without BMOE and without IHFMF on multiple samples. Specifically, without BMOE, the model's responsiveness to local details and information at different scales decreases; without inter-layer historical feature reuse, the model underutilizes effective feature information from previous layers, leading to poor boundary representation and region integrity in some samples. In contrast, the complete scheme maintains more stable segmentation output across different datasets and target morphologies, further validating the effectiveness of the balanced hybrid expert mechanism and the inter-layer historical feature fusion and reuse mechanism, as well as the technological benefits brought by their synergy.

[0073] In summary, the REFUGE2 implementation and the experimental results and performance verification on the REFUGE2, ISIC2016, and TMNIX datasets demonstrate that this application, within the framework of prompting medical image segmentation, can effectively improve the accuracy, stability, generalization ability, and practical applicability of medical image segmentation by introducing a balanced hybrid expert enhancement and inter-layer historical feature fusion and reuse mechanism.

[0074] The above embodiments are merely illustrative of the technical concept and features of this application, intended to enable those skilled in the art to understand the content of this application and implement it accordingly, and should not be construed as limiting the scope of protection of this application. It is obvious to those skilled in the art that this application is not limited to the details of the above exemplary embodiments, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of this application is defined by the appended claims rather than the foregoing description, and thus all variations falling within the meaning and scope of the equivalents of the claims are intended to be included within this application.

Claims

1. A medical image segmentation method based on inter-layer feature fusion and balanced expert suggestions, characterized in that, include: Acquire the medical image to be segmented and the prompt information, and input the medical image to be segmented into the image encoder; In at least one non-first coding layer, low-dimensional bottleneck features are extracted from the current coding layer features after multi-head attention processing and before residual connections. The routing weights generated based on the low-dimensional bottleneck features are used to weight and fuse the outputs of multiple convolutional experts to obtain enhanced bottleneck features. The enhanced bottleneck features of the preceding layers are read, and after spatial alignment, aggregation, channel projection and upsampling, historical bias features are obtained and injected into the output of the current coding layer backbone path; Perform a manifold-constrained multi-branch transformation that includes at least a residual mixing map in the current coding layer backbone path, wherein the elements in the matrix corresponding to the residual mixing map are non-negative, and the sum of the elements in each row and the sum of the elements in each column are both 1; Enhanced output features are obtained based on the main path output after injection and manifold constraint multi-branch transformation. Image embedding features are obtained based on enhanced output features, prompt information is encoded to obtain prompt embedding features, and segmentation results are obtained by decoding based on image embedding features and prompt embedding features.

2. The medical image segmentation method according to claim 1, characterized in that, The step of extracting low-dimensional bottleneck features from the current coding layer features includes: performing spatial compression and channel compression on the current coding layer features to obtain bottleneck features with both spatial resolution and channel dimension lower than those of the current coding layer features.

3. The medical image segmentation method according to claim 1, characterized in that, The multiple convolution experts include at least three convolution experts with different kernel sizes.

4. The medical image segmentation method according to claim 1, characterized in that, The aggregation includes: performing mean aggregation on the enhanced bottleneck features of the spatially aligned preceding layers to obtain historical fusion features, and performing channel projection and upsampling based on the historical fusion features.

5. The medical image segmentation method according to claim 1, characterized in that, The manifold-constrained multi-branch transformation includes: pre-mapping the multi-branch flow features of the current coding layer, obtaining the transformed features through sub-layer operations and post-mapping of the current layer, and performing residual mixing on the multi-branch flow features of the current coding layer through the residual mixing mapping.

6. The method according to claim 1, characterized in that, The prompt information includes one or more foreground points selected from the foreground pixel region of the target area.

7. The method according to claim 1, characterized in that, The segmentation result obtained by decoding based on the image embedding features and cue embedding features includes: The location encoding, the image embedding feature, and the cue embedding feature are input together into the mask decoder to obtain the decoded marker feature and the decoded image feature; Dynamic mask weights are generated based on the decoded marker features, and a predicted mask is generated based on the dynamic mask weights and the decoded image features; The predicted mask is resized to obtain the segmentation result.

8. The method according to claim 7, characterized in that, During the training phase, the loss between the predicted mask and the corresponding labeled mask ground truth is calculated. The loss calculation uses a composite loss function consisting of Dice loss and cross-entropy loss. During the inference phase, the predicted mask after size recovery is subjected to probability mapping and threshold binarization to obtain the segmentation result.

9. An electronic device, characterized in that, The device includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the electronic device to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the method according to any one of claims 1 to 8.

Images