Image segmentation method based on multimodal state space integration based on uncertainty estimation

Abstract

The present invention discloses an image segmentation method based on multimodal state space integration with uncertainty estimation. The method comprises: obtaining image data and text data corresponding to the image data; segmenting the image based on the image data and text data using a CLV-SSIF model, wherein the CLV-SSIF model includes a multimodal space integration component and a multi-expert uncertainty optimization component, wherein the multimodal space integration component includes an image encoder, a text encoder, and a multimodal interactive guided decoder; and the multi-expert uncertainty optimization component is used to generate an initial image segmentation result and expert uncertainty estimates using a multi-expert hybrid model, and then optimize the initial segmentation result based on the expert uncertainty estimates. The method of the present invention improves the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention belongs to the field of image segmentation, and in particular relates to an image segmentation method based on multimodal state space integration of uncertainty estimation. Background Art

[0002] Image segmentation is crucial in everyday applications, such as object detection, image recognition, and scene analysis. However, due to limited availability of high-quality image data and expert annotation resources, this process remains challenging. To address these issues, integrating information from multiple modalities (such as image data and descriptive text) has emerged as an effective solution. Using descriptive text to guide image segmentation can help improve the quality of datasets. In image processing, integrating multimodal information (such as images, text, and other data) is crucial for achieving accurate recognition and personalized analysis. Researchers have developed a variety of multimodal models that can integrate and interpret data from diverse sources, including vision, language, and audio, thereby enhancing image understanding and reasoning. Typical models include CLIP, ViLT, and BLIP, which have achieved promising results in integrating visual and language information. Despite significant progress, current models still struggle to effectively combine complex textual semantics with visual features, posing challenges in aligning segmentation outputs with textual descriptions. A key challenge in language-guided image segmentation is how to effectively fuse textual semantics with visual features to enhance contextual understanding and detailed representation of images. However, most existing models rely on simple feature fusion mechanisms such as dot products and cross attention, which may not be sufficient to capture complex interactions and subtle feature differences. The high foreground and background similarity commonly seen in images, especially in blurred or complex areas, may lead to inconsistencies between segmentation results and text descriptions. In addition, existing fusion of language and vision also faces challenges such as terminology and context differences affecting the alignment of text and visual features. Therefore, strengthening the interaction between text and visual features is the key to achieving more accurate and reliable segmentation. Another important challenge is how to accurately quantify and handle uncertainty in the segmentation process. Current methods often lack specialized uncertainty handling mechanisms, especially in complex areas. Due to the complex edges and fine details in ordinary images, these methods may ignore certain subtle differences, resulting in unreliable results. Summary of the Invention

[0003] In view of this, an object of the present invention is to provide an image segmentation method based on multimodal state space integration of uncertainty estimation.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] An image segmentation method based on multimodal state space integration of uncertainty estimation, comprising:

[0006] Obtaining image data and text data corresponding to the image data;

[0007] The CLV-SSIF model is used to segment images based on image data and text data. The CLV-SSIF model includes a multimodal spatial integration component and a multi-expert uncertainty optimization component.

[0008] The multimodal spatial integration component includes an image encoder, a text encoder, and a multimodal interactive guided decoder.

[0009] An image encoder, configured to receive image data and generate image features;

[0010] A text encoder, used to receive text features and generate text features;

[0011] Multimodal interactive guided decoder, used to integrate image features and text features to generate multimodal features;

[0012] The multi-expert uncertainty optimization component includes the initial segmentation module, the uncertainty estimation module and the segmentation optimization module.

[0013] The initial segmentation module is used to generate the initial segmentation results of the image using a multi-expert mixture model;

[0014] Uncertainty estimation module, used to generate expert uncertainty estimates;

[0015] The segmentation optimization module is used to optimize the initial segmentation results based on expert uncertainty estimation.

[0016] Furthermore, the multimodal spatial integration component also includes an adapter for fine-tuning text features to adapt to the outputs of image encoders at different stages.

[0017] Furthermore, the multimodal interactive guided decoder includes an image SSM module, a text SSM module, and a multimodal interaction module. The multimodal interactive guided decoder integrates image features and text features to generate multimodal features, specifically including:

[0018] Using the image SSM module, the image features are scanned through the state space to generate enhanced image features;

[0019] Using the text SSM module, the text features are scanned in the state space to generate enhanced text features;

[0020] Utilizing the multimodal interaction module and adopting the multi-head cross attention mechanism, the enhanced image features plus the image position encoding are used as the query, the enhanced text features plus the text position encoding are used as the key, and the enhanced text features are used as the value. Multi-head cross attention calculation is performed to obtain the initial multimodal features.

[0021] Furthermore, the multimodal interaction module obtains enhanced multimodal features by performing residual connections between the initial multimodal features and the enhanced image features.

[0022] Furthermore, the method also includes: after determining the enhanced multimodal features, reshaping and upsampling the multimodal features, and splicing the upsampling results with the low-level image features obtained by the image encoder through jump connections along the channel dimension, and then processing them through convolution layers and activating activation functions to generate the final multimodal features.

[0023] Furthermore, the initial segmentation module uses the multi-expert mixture model to generate the initial segmentation results of the image, including:

[0024] After receiving the multimodal features, each expert model generates a first output;

[0025] Each first output is weighted with the gating network to obtain a second output corresponding to each expert model;

[0026] The second output is weighted with the learnable mixing weight to obtain the mixed output of the multi-expert mixture model, i.e., the initial segmentation result.

[0027] Furthermore, the uncertainty estimation module generates expert uncertainty estimates specifically including:

[0028] According to the second output of each expert model and the mixed output of the multi-expert mixed model, Choquet fuzzy integral calculation is performed to obtain the mask;

[0029] According to the initial segmentation results, the area of the mask region and the area of the bounding box region in the image are determined;

[0030] Calculate the threshold based on the area of the mask region and the area of the bounding box region;

[0031] Using a threshold, the mask is converted into a binary mask, resulting in an expert uncertainty estimate.

[0032] Furthermore, the segmentation optimization module optimizes the initial segmentation results based on the expert uncertainty estimation, specifically including:

[0033] Based on the binary mask, calculate the foreground average and background average;

[0034] Set refinement parameters, and determine a refined prediction mask based on the refinement parameters, the binary mask, and the mixed output of the multi-expert mixture model;

[0035] According to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined to achieve segmentation optimization.

[0036] Furthermore, the final prediction mask is determined based on the relationship between the foreground average value, the background average value, and the mixed output, specifically including:

[0037] If the mixed output is greater than the background average and the mixed output is less than the foreground average, the refined prediction mask is used as the final prediction mask;

[0038] Otherwise, the mixed output is used as the final prediction mask.

[0039] The beneficial effects of the present invention are:

[0040] By integrating image SSM and text SSM in a multimodal interactive guided decoder, this paper fully explores the spatial and contextual information in the image. In addition, by using the cross-attention mechanism, it enhances the influence of text on visual features, improves the cross-modal fusion effect, and enhances the accuracy and interpretability of segmentation.

[0041] Then, in the subsequent segmentation stage, Choquet integral is used to optimize the uncertainty between multiple expert models, capture subtle differences, and evaluate the credibility of the model through a learnable fuzzy measure. This method enhances the reliability and detail processing capabilities of segmentation, especially in complex image areas.

[0042] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0044] Figure 1 It is the CLV-SSIF framework for language-guided image segmentation;

[0045] Figure 2 It is the structure of the multimodal interaction guided decoder MIGD. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0047] Vision-Language Models (VLMs) have attracted significant attention in recent years for their ability to integrate visual and textual data. However, existing models face challenges in integrating language and vision, such as terminology and context differences that affect the alignment of textual and visual features. Furthermore, in deep learning, uncertainty estimation is crucial for improving model reliability and trustworthiness, particularly in general image processing, where it helps identify key regions and supports clinicians. Uncertainty can be categorized into two types: aleatoric uncertainty caused by data noise and epistemic uncertainty caused by model knowledge limitations. In general image segmentation, uncertainty helps assess the accuracy of pixel classification and guides clinical decision-making by locating unreliable predictions. Existing uncertainty estimation methods include Bayesian neural networks, Monte Carlo dropout, and ensemble techniques. For example, UR-SAM, an uncertainty-corrected SAM (SegmentAnything Model) framework, improves segmentation performance by generating and correcting uncertainty maps without requiring additional training. Existing active learning based on selective uncertainty improves accuracy with fewer labeled samples by focusing on key regions. However, despite the huge potential of multimodal data (such as integrating radiographic reports and image data) to improve diagnostic accuracy and model robustness, it is currently underutilized.

[0048] To this end, the method proposed in this paper further improves cross-modal alignment and fusion techniques to more efficiently integrate multimodal data, enabling the model to prioritize key information that enhances segmentation accuracy.

[0049] An image segmentation method based on multimodal state space integration of uncertainty estimation may specifically include:

[0050] Obtaining image data and text data corresponding to the image data;

[0051] Using the CLV-SSIF model, the image is segmented based on the image data and text data. Figure 1 It is a CLV-SSIF framework for language-guided image segmentation. Figure 1 As shown in Figure 2, the CLV-SSIF model includes a multimodal spatial integration component and a multi-expert uncertainty optimization component. That is, in the CLV-SSIF framework, multimodal spatial integration is performed in the first stage, and multi-expert uncertainty optimization is performed in the second stage.

[0052] The multimodal spatial integration component includes an image encoder, a text encoder, and a multimodal interactive guided decoder.

[0053] An image encoder, configured to receive image data and generate image features;

[0054] A text encoder, used to receive text features and generate text features;

[0055] A multimodal interactive guided decoder is used to integrate image features and text features to generate multimodal features.

[0056] The multi-expert uncertainty optimization component includes the initial segmentation module, the uncertainty estimation module and the segmentation optimization module.

[0057] The initial segmentation module is used to generate the initial segmentation results of the image using a multi-expert mixture model;

[0058] Uncertainty estimation module, used to generate expert uncertainty estimates;

[0059] The segmentation optimization module is used to optimize the initial segmentation results based on expert uncertainty estimation.

[0060] The image encoder can use a ConvNeXt convolution-based encoder to extract features from the image. At the same time, the pre-trained weights of ImageNet can be used to accelerate convergence.

[0061] The text encoder can use the pre-trained BERT model. Through the collaborative work of the image encoder and the text encoder, image and text features can be effectively integrated.

[0062] For example, given an input image and the and medical texts Among them, H represents the image height, W represents the image width, and L represents the number of tokens after tokenizing the text. represents the set of real numbers.

[0063] f i =Image_Encoder(x i ), (1)

[0064] f t =Text_Encoder(x t ), (2)

[0065] in, Represents the image features generated by the image encoder Image_Encoder, and C represents the number of channels; Represents the text features extracted by the text encoder Text_Encoder, and D represents the feature dimension.

[0066] For the image encoder based on ConvNeXt convolution, it consists of four stages. Each stage generates image features of different scales, such as Figure 1 The image feature 1 shown is image feature 1 (i.e., the output of the first layer of the image encoder), image feature 2 (i.e., the output of the second layer of the image encoder), image feature 3 (i.e., the output of the third layer of the image encoder), and image feature 4 (i.e., the output of the fourth layer of the image encoder). Therefore, in order to align with image features at different stages (different scales), the multimodal space integration component also includes an adapter for fine-tuning the text features to adapt to the outputs of image encoders at different stages. For this embodiment, there are 3 adapters, which adjust the text tokens to fine-tune the text features. Among them, the adapter may include a dimensionality reduction layer, an activation function layer, and a dimensionality increase layer.

[0067]

[0068]

[0069] in, C1, C2 and C represent the image features extracted from different layers (layer 1, layer 2, ..., layer n) of the image encoder. n Represents the number of channels of different layer features. Represents the text features after fine-tuning by the corresponding adapter, n represents the number of layers of image features output by the image encoder; n-1 represents the number of layers of text features after fine-tuning by the adapter (one layer less than the total number of image features).

[0070] After acquiring the image features and text features, these features are processed by a multimodal interactive guided decoder MIGD at each scale, thereby interacting the output of the image encoder with the output of the corresponding Adapter. Specifically, each decoder MIGD requires three inputs, namely the text features fine-tuned by the adapter, the image features of the previous layer, and the image features of the current layer. Figure 1 For the first decoder MIGD, image feature 4 (i.e., the image feature of the previous layer) and image feature 3 (i.e., the image feature of this layer) are required; for the second decoder MIGD, the output of the first decoder MIGD is used as the image feature of the previous layer, and image feature 2 is used as the image feature of this layer; for the third decoder, the output of the second decoder MIGD is used as the image feature of the previous layer, and image feature 1 is used as the image feature of this layer. This process can be expressed as:

[0071]

[0072] in, Represents different multimodal features generated by MIGD.

[0073] These features are then fused into a joint embedding space to align and integrate cross-modal information, and finally upsampled before entering the next stage.

[0074] Figure 2 This is the structure diagram of the multimodal interactive guided decoder MIGD, as shown in Figure 2 As shown in Figure 2, MIGD includes an image SSM module, a text SSM module, and a multimodal interaction module.

[0075] The multimodal interactive guided decoder integrates image features and text features to generate multimodal features, including:

[0076] Using the image SSM module, the image features are scanned through the state space to generate enhanced image features;

[0077] The text SSM module is used to scan the text features in state space to generate enhanced text features.

[0078] Conventional state space models (SSMs) map input sequences to output sequences through hidden states. In the present invention, the image SSM module (also referred to as the "image selective scanning module") and the text SSM module (also referred to as the "text selective scanning module") are both selective state space models. The image SSM module includes an image state space model, and the text SSM module includes a text state space model. Text features and image features are learned using a text state space model (also referred to as Text-SSM) and an image state space model (also referred to as Image-SSM), respectively, to capture long-range dependencies in the two modal data and enhance feature representation.

[0079] The present invention processes image feature blocks in four directions through Mamba's two-dimensional selective scanning (written as "SS2D"), namely from left to right, from right to left, from top to bottom, and from bottom to top, thereby enhancing the global receptive field of the model. Since image features are 2D (i.e., two-dimensional) and text features are 1D (i.e., one-dimensional), the image feature f is processed in four directions: from left to right, from right to left, from top to bottom, and from bottom to top. i Using SS2D (i.e. Figure 2 The image state space model in is processed, and the text feature only needs one direction, so f t Only S6 (i.e. Figure 2 The text state space model in the text state space model is enhanced, and the above process is expressed as follows:

[0080]

[0081]

[0082] Specifically, refer to Figure 2 , input SS2D front image feature f i It is processed by layer normalization (LayerNorm), and then input into the state space scan SS2D and also needs normalization (LayerNorm). Finally, the jump connection is used to retain the original normalized feature LayerNorm (f i ), thereby obtaining an enhanced image Text features Linear projection (Liner) is required before inputting S6. Liner is a fully connected neural network used for linear projection. Then, LayerNorm is used and the state space is scanned by S6. Finally, after normalizing the features, feature addition is used to retain the original text features of a certain scale to obtain enhanced text. The scale factor ω is a learnable parameter and is optimized over iterations.

[0083] After processing the image and text, the enhanced image and enhanced text features The input is fed into the multimodal interaction module. In this module, MIGD employs a cross-modal multi-head cross-attention mechanism to capture the correlation between different modalities by calculating the similarity matrix between text and image. Specifically, MIGD uses image features as queries and text features as keys and values, and calculates attention weights to enhance image regions that are highly relevant to the text description.

[0084] Specifically, the multimodal interaction module is used, and a multi-head cross-attention mechanism is adopted. The enhanced image features plus the image position encoding are used as the query, the enhanced text features plus the text position encoding are used as the key, and the enhanced text features are used as the value. Multi-head cross-attention calculation is performed to obtain the initial multimodal features.

[0085] The image features obtained by scanning the state space are recorded as (i.e., the enhanced image features obtained in Formula 6), the text features are recorded as (i.e., the enhanced text features obtained in Formula 6), then the query f in the multimodal cross attention mechanism q 、key f k and the value f v It can be expressed as: Among them, P i Represents the position code of the image; P t Represents the positional encoding of the text.

[0086] In some embodiments, the multimodal interaction module further obtains enhanced multimodal features by performing a residual connection between the initial multimodal features and the enhanced image features. Then the enhanced multimodal features f m It can be expressed as:

[0087]

[0088] Among them, MHCA represents the multi-head cross attention algorithm, β represents the learnable parameters, which is combined with the initial features of the image (i.e., enhanced image features) The residual connection, multiplied by the learnable parameter β, enhances its expressive power. The result of MHCA is obtained by concatenating the outputs of multiple attention heads and multiplying them by the learnable output weight matrix W. 0 form.

[0089] in, i ranges from 1 to h, where h represents the number of heads of multi-head attention, and d k represents the scaling factor, the superscript T represents the transpose, and Represents the learnable weight matrix corresponding to each attention head, which is used to transform the query, key, and value features. The attention scores are obtained by calculating the scaled dot product of the transformed query and key features and performing a softmax operation. These scores are then used to weight the transformed value features.

[0090] After determining the enhanced multimodal features f m After that, the multimodal features are reshaped (i.e., Reshape in Formula 8) and upsampled (i.e., Upsample in Formula 8), and the upsampled results are Low-level image features obtained through skip connections with the image encoder Splicing is performed along the channel dimension (i.e., [,] in Formula 9), ω represents the scale factor, and then processed by the convolution layer (i.e., Conv in Formula 9) and the activation function (i.e., the ReL activation function in Formula 9) to generate the final multimodal features It can be expressed as:

[0091]

[0092]

[0093] After the final multimodal features are determined, the second stage, the multi-expert uncertainty optimization stage, is entered. Figure 1 Provide explanation.

[0094] In the second stage, the Multi-Expert Uncertainty Optimization (MEUR) component employs advanced methods such as Choquet Integral (CI) to assess the uncertainty of the multi-expert model's outputs. MEUR improves segmentation reliability by capturing subtle variations and consistencies between expert predictions. CI methods specifically highlight differences between expert outputs, providing a robust quantification of segmentation uncertainty. This refinement process not only improves segmentation accuracy but also facilitates detailed evaluation of model outputs.

[0095] The multi-expert uncertainty optimization component includes the initial segmentation module, the uncertainty estimation module and the segmentation optimization module.

[0096] The initial segmentation module is used to generate the initial segmentation results of the image using a multi-expert mixture model;

[0097] Uncertainty estimation module, used to generate expert uncertainty estimates;

[0098] The segmentation optimization module is used to optimize the initial segmentation results based on expert uncertainty estimation.

[0099] The Mixture of Experts (MoE) model is a neural network structure that combines the outputs of multiple expert models (sub-models) by weighting them. The key idea is to introduce a gating network to dynamically select the contribution of each expert model, thereby improving the overall performance of the model.

[0100] Specifically, the initial segmentation module uses a multi-expert mixture model to generate the initial segmentation results of the image, including:

[0101] After receiving the input multimodal features f o Then, input x=f o , each expert model E i Generate a first output e i , that is, e i =E i (x);

[0102] Each first output is weighted with the gating network, and the weight of the gating network is g i =G i (x), and obtain the second output e corresponding to each expert model i ',Right now

[0103] The second output is mixed with the learnable weight Φ i Perform weighting to obtain the mixed output p of the multi-expert mixture model, That is the initial segmentation result.

[0104] Where M represents the number of expert models.

[0105] The learnable weight g i satisfy: And g i ≥0.

[0106] Subsequently, in order to further improve the performance of the model, the present invention uses a further uncertainty refinement mechanism to improve MoE and enhance its expressive power.

[0107] Choquet integral is a widely used method for fuzzy measurement, with wide applications in decision theory, image processing and fuzzy systems. By combining the significance of input features and their weights, Choquet integral provides a comprehensive evaluation of multiple features. This method facilitates the integration of fuzzy sets and more accurately captures the interactions and dependencies between variables. Choquet integral is crucial for combining feature weights and values, effectively managing uncertainty and ambiguity.

[0108] In some embodiments, the uncertainty estimation module generates the expert uncertainty estimate by:

[0109] According to the second output of each expert model and the mixed output of the multi-expert mixture model (for example, the second output and the mixed output of all expert models can be spliced into a feature set), the Choquet fuzzy integral calculation is performed to obtain the mask m. The mask matrix is expressed as follows: Among them, f σ(i) It represents the eigenvalue of the i-th expert model obtained by sorting the eigenvalues of the expert models in ascending order, and σ(i) represents the index after sorting in descending order according to the eigenvalues; represents the cumulative weight of the expert model after sorting from 0 to i, w (j) is the learnable fuzzy measure value of the expert model;

[0110] According to the initial segmentation results, determine the area A of the mask area in the image mask and the area of the bounding box region A bbox ;

[0111] According to the area of the mask region and the area of the bounding box region, a threshold θ is calculated. For example, θ can be any suitable value. In some embodiments, it can be

[0112] Using the threshold θ, the mask is converted into a binary mask, that is, the pixel feature m corresponding to each pixel in the mask is converted into i Compared with the threshold θ, if m i If the value of the pixel is not less than the threshold θ, the pixel feature corresponding to the pixel is set to 1.0; otherwise, the pixel feature corresponding to the pixel is set to 0, thereby obtaining the expert uncertainty estimate. The binary mask is represented as:

[0113]

[0114] Furthermore, the segmentation optimization module optimizes the initial segmentation results based on the expert uncertainty estimation, specifically including:

[0115] According to the binary mask, calculate the foreground mean p f and the background average value p b , the specific formula can be expressed as:

[0116]

[0117]

[0118] Formula (11) represents the sum of the positions where the pixel is greater than or equal to 0.5, and then divided by the position where the pixel is greater than or equal to 0.5, rounded up to 1 (that is, the denominator represents the total number of pixels where the pixel is greater than or equal to 0.5), which represents the foreground average value. The background average value in formula (12) is similar. N represents the number of pixels; p i Represents the pixel value of the i-th pixel in the mixed output p of the multi-expert mixture model.

[0119] Next, the refinement parameter λ is set, and the refined prediction mask p is determined based on the refinement parameter λ, the binary mask, and the mixed output of the multi-expert mixture model. r , expressed as: p r =p+λ·m;

[0120] According to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined to achieve segmentation optimization.

[0121] In some embodiments, determining a final prediction mask based on a relationship between the foreground average value, the background average value, and the mixed output specifically includes:

[0122] If the mixed output is greater than the background average and the mixed output is less than the foreground average, the refined prediction mask is used as the final prediction mask;

[0123] Otherwise, the mixed output is used as the final prediction mask, that is, it can be expressed as:

[0124]

[0125] Among them, (p r >p b )∧(p r <p f ) indicates p r >p bEstablished and p r <p f Established.

[0126] The proposed CLV-SSIF framework is a novel language-guided medical image segmentation framework. By integrating knowledge from the language and vision domains, CLV-SSIF enhances image analysis capabilities. MIGD promotes interaction between the Text-SSM and Image-SSM modules, utilizing cross-attention to extract contextual information, thereby improving segmentation accuracy. Furthermore, the MEUR component employs the CI method to quantify segmentation uncertainty, thereby enhancing the reliability and detail of the results. Comparison results on two different datasets consistently demonstrate that the proposed method performs well in achieving high-precision segmentation. This advance provides a promising direction for future research and applications in medical imaging.

[0127] The advantages of the present invention are described below through specific experiments.

[0128] The experiments in this paper were implemented in the PyTorch framework. Training was performed on a high-performance device equipped with an NVIDIA GeForce RTX3090 GPU, which effectively managed the computational requirements. The initial learning rate was set to 3×10 -4 ,

[0129] A cosine annealing strategy is used as the learning rate scheduler to improve convergence. The AdamW optimizer is used to dynamically adjust the learning rate to enhance the optimization effect. The default batch size is 32, and all input images are resized to 224×224 pixels to ensure the consistency of the dataset. The number of experts is set to 4. Experiments are conducted on two medical datasets: the QaTa-COV19 dataset and the MosMedData+ dataset. Two standard metrics are used for evaluation: Dice score and mean intersection over union (mIoU), which are commonly used to evaluate the accuracy in object contouring tasks. The model is trained for 200 epochs on the QaTa-COV19 dataset and 400 epochs on the MosMedData+ dataset to enhance the robustness and comprehensive learning of the model.

[0130] To investigate the performance of various current methods and our proposed CLV-SSIF approach in medical image segmentation, we conducted benchmark tests using different datasets. Table 1 shows the medical image segmentation experiments conducted on the QaTa-COV19 and MosMedData+ datasets, using CNN-based methods (such as U-Ne, U-Net++, and AttUNet) as well as hybrid methods (such as TransUNet and Swin Unet). In addition, this experiment also compares the latest multimodal segmentation models, including LViT, RecLMIS, LanGuideMedSeg, and the proposed CLV-SSIF.

[0131] Table 1 Results of image segmentation using different methods

[0132]

[0133]

[0134] It should be noted that "×" in Table 1 means that text is not included; "√" means that text is included. The experimental results in Table 1 show that CLV-SSIF outperforms these state-of-the-art methods. Specifically, CLV-SSIF achieved Dice coefficients of 90.80% and 79.46% and mIoU scores of 83.15% and 65.92% on the QaTa-COV19 and MosMedData+ datasets, respectively. These results are significantly improved over traditional CNN architectures and other hybrid models, demonstrating the effectiveness of integrating multimodal inputs. It is worth noting that CLV-SSIF maintains high accuracy and achieves the lowest FLOPs (11.1) with a moderate number of parameters (60 million), demonstrating its computational efficiency.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. An image segmentation method based on multimodal state space integration of uncertainty estimation, characterized in that: include: Obtaining image data and text data corresponding to the image data; The CLV-SSIF model is used to segment images based on image data and text data. The CLV-SSIF model includes a multimodal spatial integration component and a multi-expert uncertainty optimization component. The multimodal spatial integration component includes an image encoder, a text encoder, and a multimodal interactive guided decoder. An image encoder, configured to receive image data and generate image features; A text encoder, used to receive text features and generate text features; Multimodal interactive guided decoder, used to integrate image features and text features to generate multimodal features; The multi-expert uncertainty optimization component includes the initial segmentation module, the uncertainty estimation module and the segmentation optimization module. The initial segmentation module is used to generate the initial segmentation results of the image using a multi-expert mixture model; Uncertainty estimation module, used to generate expert uncertainty estimates; Segmentation optimization module, used to optimize the initial segmentation results based on expert uncertainty estimation; The multi-expert mixture model includes multiple expert models and a gating network. The initial segmentation module uses the multi-expert mixture model to generate an initial segmentation result for the image, specifically including: After receiving the multimodal features, each expert model generates a first output; Each first output is weighted with the gating network to obtain a second output corresponding to each expert model; The second output is weighted with the learnable mixing weight to obtain the mixed output of the multi-expert mixture model, that is, the initial segmentation result; The uncertainty estimation module generates expert uncertainty estimates specifically including: According to the second output of each expert model and the mixed output of the multi-expert mixed model, Choquet fuzzy integral calculation is performed to obtain the mask; According to the initial segmentation results, the area of the mask region and the area of the bounding box region in the image are determined; Calculate the threshold based on the area of the mask region and the area of the bounding box region; Using a threshold, the mask is converted into a binary mask to obtain an expert uncertainty estimate; The segmentation optimization module optimizes the initial segmentation results based on the expert uncertainty estimate, specifically including: Based on the binary mask, calculate the foreground average and background average; Set refinement parameters, and determine a refined prediction mask based on the refinement parameters, the binary mask, and the mixed output of the multi-expert mixture model; According to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined to achieve segmentation optimization.

2. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 1, characterized in that: The multimodal spatial integration component also includes adapters for fine-tuning text features to the outputs of image encoders at different stages.

3. The image segmentation method based on multimodal state space integration based on uncertainty estimation according to claim 1, characterized in that: The multimodal interactive guided decoder includes an image SSM module, a text SSM module, and a multimodal interaction module. The multimodal interactive guided decoder integrates image features and text features to generate multimodal features, specifically including: Using the image SSM module, the image features are scanned through the state space to generate enhanced image features; Using the text SSM module, the text features are scanned in the state space to generate enhanced text features; Utilizing the multimodal interaction module and adopting the multi-head cross attention mechanism, the enhanced image features plus the image position encoding are used as the query, the enhanced text features plus the text position encoding are used as the key, and the enhanced text features are used as the value. Multi-head cross attention calculation is performed to obtain the initial multimodal features.

4. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 3, characterized in that: The multimodal interaction module also obtains enhanced multimodal features by performing residual connections between the initial multimodal features and the enhanced image features.

5. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 4, characterized in that: The method also includes: after determining the enhanced multimodal features, reshaping and upsampling the multimodal features, and splicing the upsampling results with the low-level image features obtained by the image encoder through jump connections along the channel dimension, and then processing them through convolution layers and activating activation functions to generate the final multimodal features.

6. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 1, characterized in that: The final prediction mask is determined based on the relationship between the foreground average value, the background average value and the mixed output, including: If the mixed output is greater than the background average and the mixed output is less than the foreground average, the refined prediction mask is used as the final prediction mask; Otherwise, the mixed output is used as the final prediction mask.

Images