Multi-modal state space integrated image segmentation method based on uncertainty estimation

Abstract

The invention discloses a multi-modal state space integrated image segmentation method based on uncertainty estimation. The method comprises the following steps: acquiring image data and text data corresponding to the image data; a CLV-SSIF model is utilized, according to the image data and the text data, the image is segmented, the CLV-SSIF model comprises a multi-modal space integration assembly and a multi-expert uncertainty optimization assembly, and the multi-modal space integration assembly comprises an image encoder, a text encoder and a multi-modal interactive guide decoder; and the multi-expert uncertainty optimization component is used for generating an initial segmentation result of the image and generating expert uncertainty estimation by using the multi-expert hybrid model, and then optimizing the initial segmentation result according to the expert uncertainty estimation. According to the method, the accuracy of image segmentation is improved.

Description

Technical Field

[0001] The invention belongs to the field of image segmentation, and in particular relates to an image segmentation method based on multi-modal state space integration of uncertainty estimation. Background Art

[0002] Image segmentation is very important in daily applications, such as object detection, image recognition, and scene analysis. However, this process is still challenging due to the limited resources of high-quality image data and expert-level annotation. To address these problems, integrating information from multiple modalities (such as image data and descriptive text) has become an effective solution. Using descriptive text to guide image segmentation can help improve the quality of the dataset. In image processing, integrating multimodal information (such as images, text, and other data) is of great significance for achieving accurate recognition and personalized analysis. Researchers have developed a variety of multimodal models that can integrate and interpret data from different sources such as vision, language, and audio, thereby enhancing the understanding and reasoning ability of images. Typical models include CLIP, ViLT, and BLIP, which have achieved good results in integrating visual and language information. Despite significant progress, current models still have difficulties in effectively combining complex text semantic information with visual features, which poses a challenge to aligning segmentation output with text descriptions. In language-guided image segmentation, a key challenge is how to effectively fuse text semantic information with visual features to enhance the contextual understanding and detailed expression of images. However, most existing models rely on simple feature fusion mechanisms such as dot products and cross attention, which may not be sufficient to capture complex interactions and subtle feature differences. The high foreground-background similarity common in images, especially in blurred or complex areas, may lead to inconsistencies between segmentation results and text descriptions. In addition, the existing fusion of language and vision also faces challenges such as terminology and context differences affecting the alignment of text and visual features. Therefore, strengthening the interaction between text and visual features is the key to achieving more accurate and reliable segmentation. Another important challenge is how to accurately quantify and handle uncertainty in the segmentation process. Current methods often lack dedicated uncertainty handling mechanisms, especially in complex areas. Due to the presence of complex edges and fine details in ordinary images, these methods may ignore certain subtle differences, resulting in unreliable results. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an image segmentation method based on multimodal state space integration of uncertainty estimation.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] An image segmentation method based on multimodal state space integration of uncertainty estimation, comprising:

[0006] Acquire image data and text data corresponding to the image data;

[0007] The CLV-SSIF model is used to segment images based on image data and text data. The CLV-SSIF model includes a multimodal spatial integration component and a multi-expert uncertainty optimization component.

[0008] The multimodal spatial integration component includes an image encoder, a text encoder, and a multimodal interactive guided decoder.

[0009] An image encoder, for receiving image data and generating image features;

[0010] A text encoder, used to receive text features and generate text features;

[0011] Multimodal interactive guided decoder, used to integrate image features and text features to generate multimodal features;

[0012] The multi-expert uncertainty optimization component includes an initial segmentation module, an uncertainty estimation module, and a segmentation optimization module.

[0013] An initial segmentation module is used to generate an initial segmentation result of an image using a multi-expert mixture model;

[0014] Uncertainty estimation module, used to generate expert uncertainty estimates;

[0015] The segmentation optimization module is used to optimize the initial segmentation results based on the expert uncertainty estimation.

[0016] Furthermore, the multimodal spatial integration component also includes an adapter for fine-tuning the text features to adapt to the outputs of the image encoders at different stages.

[0017] Furthermore, the multimodal interactive guided decoder includes an image SSM module, a text SSM module and a multimodal interaction module. The multimodal interactive guided decoder integrates image features and text features to generate multimodal features, specifically including:

[0018] Using the image SSM module, the image features are scanned through the state space to generate enhanced image features;

[0019] Using the text SSM module, the text features are scanned in state space to generate enhanced text features;

[0020] By utilizing the multimodal interaction module and adopting the multi-head cross-attention mechanism, the enhanced image features plus the image position encoding are used as the query, the enhanced text features plus the text position encoding are used as the key, and the enhanced text features are used as the value. Multi-head cross-attention calculation is performed to obtain the initial multimodal features.

[0021] Furthermore, the multimodal interaction module obtains enhanced multimodal features by performing residual connection between the initial multimodal features and the enhanced image features.

[0022] Furthermore, the method also includes: after determining the enhanced multimodal features, reshaping and upsampling the multimodal features, and splicing the upsampling results with the low-level image features obtained by the image encoder through jump connections along the channel dimension, and then processing through the convolution layer and activating the activation function to generate the final multimodal features.

[0023] Furthermore, the multi-expert mixture model includes multiple expert models and a gating network. The initial segmentation module uses the multi-expert mixture model to generate the initial segmentation results of the image, specifically including:

[0024] After receiving the multimodal features, each expert model generates a first output;

[0025] Each first output is weighted with the gating network to obtain a second output corresponding to each expert model;

[0026] The second output is weighted with the learnable mixing weight to obtain the mixed output of the multi-expert mixture model, i.e., the initial segmentation result.

[0027] Furthermore, the uncertainty estimation module generates expert uncertainty estimates specifically including:

[0028] According to the second output of each expert model and the mixed output of the multi-expert mixed model, Choquet fuzzy integral calculation is performed to obtain a mask;

[0029] According to the initial segmentation result, determine the area of ​​the mask region and the area of ​​the bounding box region in the image;

[0030] Calculate the threshold based on the area of ​​the mask region and the area of ​​the bounding box region;

[0031] Using a threshold, the mask is converted into a binary mask to obtain an expert uncertainty estimate.

[0032] Furthermore, the segmentation optimization module optimizes the initial segmentation results according to the expert uncertainty estimation, specifically including:

[0033] Based on the binary mask, calculate the foreground average and background average;

[0034] Setting refinement parameters, and determining a refined prediction mask according to the refinement parameters, the binary mask, and the mixed output of the multi-expert mixture model;

[0035] According to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined to achieve segmentation optimization.

[0036] Furthermore, according to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined, which specifically includes:

[0037] If the mixed output is greater than the background average and the mixed output is less than the foreground average, the refined prediction mask is used as the final prediction mask;

[0038] Otherwise, the mixed output is used as the final prediction mask.

[0039] The beneficial effects of the present invention are:

[0040] The present invention integrates image SSM and text SSM in a multimodal interactive guided decoder, fully explores the spatial and contextual information in the image, and uses the cross-attention mechanism to enhance the influence of text on visual features, improve the cross-modal fusion effect, and enhance the accuracy and interpretability of segmentation.

[0041] Then, in the subsequent segmentation stage, Choquet integral is used to optimize the uncertainty between multiple expert models, capture subtle differences, and evaluate the credibility of the model through a learnable fuzzy measure. This method enhances the reliability and detail processing capabilities of segmentation, especially in complex image areas.

[0042] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0044] Figure 1 It is the CLV-SSIF framework for language-guided image segmentation;

[0045] Figure 2 It is the structure of the multimodal interaction guided decoder MIGD. DETAILED DESCRIPTION

[0046] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0047] In recent years, vision-language models (VLMs) have attracted much attention due to their ability to integrate visual and textual data. However, existing models also face challenges in the integration of language and vision, such as terminology and context differences affecting the alignment of text and visual features. Moreover, in deep learning, uncertainty estimation is crucial to improving the reliability and credibility of the model, especially in general image processing, which helps to identify key areas and provide support for clinicians. Uncertainty can be divided into two categories: aleatoric uncertainty caused by data noise and epistemic uncertainty caused by model knowledge limitations. In general image segmentation, uncertainty helps evaluate the accuracy of pixel classification and guides clinical decision-making by locating unreliable predictions. Existing uncertainty estimation methods include Bayesian neural networks, Monte Carlo Dropout, and integration techniques. For example, the existing UR-SAM, an uncertainty-corrected SAM (SegmentAnything Model) framework, improves segmentation results by generating and correcting uncertainty maps without additional training. Existing active learning based on selective uncertainty improves accuracy with fewer labeled samples by focusing on key areas. However, despite the huge potential of multimodal data (such as integrating radiographic reports and image data) in improving diagnostic accuracy and model robustness, it is currently underutilized.

[0048] To this end, the method proposed in the present invention further improves the cross-modal alignment and fusion technology, thereby integrating multimodal data more efficiently and enabling the model to prioritize key information to enhance segmentation accuracy.

[0049] An image segmentation method based on multimodal state space integration of uncertainty estimation may specifically include:

[0050] Acquire image data and text data corresponding to the image data;

[0051] Using the CLV-SSIF model, the image is segmented based on the image data and text data. Figure 1 It is a CLV-SSIF framework for language-guided image segmentation. Figure 1 As shown in Figure 1, the CLV-SSIF model includes a multimodal spatial integration component and a multi-expert uncertainty optimization component. That is, in the CLV-SSIF framework, multimodal spatial integration is performed in the first stage, and multi-expert uncertainty optimization is performed in the second stage.

[0052] The multimodal spatial integration component includes an image encoder, a text encoder, and a multimodal interactive guided decoder.

[0053] An image encoder, for receiving image data and generating image features;

[0054] A text encoder, used to receive text features and generate text features;

[0055] A multimodal interactive guided decoder is used to integrate image features and text features to generate multimodal features.

[0056] The multi-expert uncertainty optimization component includes an initial segmentation module, an uncertainty estimation module, and a segmentation optimization module.

[0057] An initial segmentation module is used to generate an initial segmentation result of an image using a multi-expert mixture model;

[0058] Uncertainty estimation module, used to generate expert uncertainty estimates;

[0059] The segmentation optimization module is used to optimize the initial segmentation results based on the expert uncertainty estimation.

[0060] The image encoder can use a ConvNeXt convolution-based encoder to extract features from the image. At the same time, the pre-trained weights of ImageNet can be used to accelerate convergence.

[0061] The text encoder can use the pre-trained BERT model. Through the collaborative work of the image encoder and the text encoder, the image and text features can be effectively integrated.

[0062] For example, given an input image and the and medical texts Among them, H represents the image height, W represents the image width, and L represents the number of tokens after the text is tokenized. Represents the set of real numbers.

[0063] f i =Image_Encoder(x i ), (1)

[0064] f t =Text_Encoder(x t ), (2)

[0065] in, Represents the image features generated by the image encoder Image_Encoder, and C represents the number of channels; Represents the text features extracted by the text encoder Text_Encoder, and D represents the feature dimension.

[0066] For the image encoder based on ConvNeXt convolution, it includes four stages, each stage will generate image features of different scales, such as Figure 1 The image feature 1 (i.e., the output of the first layer of the image encoder), image feature 2 (i.e., the output of the second layer of the image encoder), image feature 3 (i.e., the output of the third layer of the image encoder), and image feature 4 (i.e., the output of the fourth layer of the image encoder) are shown. Therefore, in order to align with image features at different stages (different scales), the multimodal space integration component also includes an adapter for fine-tuning the text features to adapt to the outputs of image encoders at different stages. For this embodiment, there are 3 adapters, which adjust the text tokens to fine-tune the text features respectively. Among them, the adapter may include a dimensionality reduction layer, an activation function layer, and a dimensionality increase layer.

[0067]

[0068]

[0069] in, C1, C2 and C represent the image features extracted from different layers (layer 1, layer 2, ..., layer n) of the image encoder. n They represent the number of channels of different layer features. It represents the text features after fine-tuning by the corresponding adapter, n represents the number of layers of image features output by the image encoder; n-1 represents the number of layers of text features after fine-tuning by the adapter (one layer less than the total number of image features).

[0070] After acquiring the image features and text features, these features are processed by a multimodal interactive guided decoder MIGD at each scale, so that the output of the image encoder interacts with the output of the corresponding Adapter. Specifically, each decoder MIGD requires three inputs, namely the text features fine-tuned by the adapter, the image features of the previous layer, and the image features of the current layer. Figure 1 For the first decoder MIGD, image feature 4 (i.e., image feature of the previous layer) and image feature 3 (i.e., image feature of this layer) are required; for the second decoder MIGD, the output of the first decoder MIGD is used as the image feature of the previous layer, and image feature 2 is used as the image feature of this layer; for the third decoder, the output of the second decoder MIGD is used as the image feature of the previous layer, and image feature 1 is used as the image feature of this layer. The process can be expressed as:

[0071]

[0072] in, Represents different multimodal features generated by MIGD.

[0073] These features are then fused into a joint embedding space to align and integrate cross-modal information and finally upsampled before entering the next stage.

[0074] Figure 2 It is the structure diagram of the multimodal interactive guided decoder MIGD, as shown in Figure 2 As shown in Figure 1, MIGD includes an image SSM module, a text SSM module, and a multimodal interaction module.

[0075] The multimodal interactive guided decoder integrates image features and text features to generate multimodal features, including:

[0076] Using the image SSM module, the image features are scanned through the state space to generate enhanced image features;

[0077] Using the text SSM module, the text features are scanned in state space to generate enhanced text features.

[0078] Conventional state space models (SSMs) map input sequences to output sequences through hidden states. In the present invention, the image SSM module (also referred to as "image selective scanning module") and the text SSM module (also referred to as "text selective scanning module") are both selective state space models. The image SSM module includes an image state space model, and the text SSM module includes a text state space model. Text features and image features are learned through a text state space model (also referred to as Text-SSM) and an image state space model (also referred to as Image-SSM), respectively, to capture long-distance dependencies in the two modal data and enhance feature representation.

[0079] The present invention processes image feature blocks in four directions through Mamba's two-dimensional selective scanning (written as "SS2D"), namely, from left to right, from right to left, from top to bottom, and from bottom to top, thereby enhancing the global receptive field of the model. Since image features are 2D (i.e., two-dimensional) and text features are 1D (i.e., one-dimensional), the image feature f i Use SS2D (i.e. Figure 2 The image state space model in is used for processing, and the text feature only needs one direction, so f t Only S6 (i.e. Figure 2 The text state space model in the text state space model is enhanced, and the above process is expressed as follows:

[0080]

[0081]

[0082] Specifically, refer to Figure 2 , input SS2D front image feature f i It is processed by layer normalization (LayerNorm), and then input into the state space scan SS2D and also needs normalization (LayerNorm). Finally, the jump connection is used to retain the original normalized feature LayerNorm (f i ), thereby obtaining an enhanced image Text Features Linear projection (Liner) is required before inputting S6. Liner is a fully connected neural network for linear projection. Then, LayerNorm is used to scan the state space S6. Finally, after normalizing the features, feature addition is used to retain the original text features of a certain scale to obtain enhanced text. The scale factor ω is a learnable parameter and is optimized over iterations.

[0083] After processing the image and text, the enhanced image and enhanced text features It is input into the multimodal interaction module. In this module, MIGD adopts a cross-modal multi-head cross-attention mechanism to capture the correlation between different modalities by calculating the similarity matrix between text and image. Specifically, MIGD uses image features as queries and text features as keys and values, and enhances image areas that are highly related to text descriptions through attention weight calculation.

[0084] Specifically, the multimodal interaction module is used, and a multi-head cross-attention mechanism is adopted. The enhanced image features plus the image position encoding are used as the query, the enhanced text features plus the text position encoding are used as the key, and the enhanced text features are used as the value. Multi-head cross-attention calculation is performed to obtain the initial multimodal features.

[0085] The image features obtained by scanning the state space are recorded as (i.e., the enhanced image features obtained in Formula 6), and the text features are recorded as (i.e., the enhanced text features obtained in Formula 6), then the query f in the multimodal cross attention mechanism q , key f k and the value f v It can be expressed as: Among them, P i Represents the position code of the image; P t Represents the positional encoding of the text.

[0086] In some embodiments, the multimodal interaction module further obtains enhanced multimodal features by performing a residual connection between the initial multimodal features and the enhanced image features. Then the enhanced multimodal features f m It can be expressed as:

[0087]

[0088] Among them, MHCA represents the multi-head cross attention algorithm, β represents the learnable parameters, and the initial features of the image (i.e., the enhanced image features) are combined with the The residual connection, multiplied by the learnable parameter β, enhances its expressive power. The result of MHCA is obtained by concatenating the outputs of multiple attention heads and multiplying them by the learnable output weight matrix W. 0 form.

[0089] in, i ranges from 1 to h, where h represents the number of heads of multi-head attention, and d k represents the scaling factor, the superscript T represents the transpose, and Represents the learnable weight matrix corresponding to each attention head, used to transform the query, key, and value features. The attention scores are obtained by calculating the scaled dot product of the transformed query and key features and performing a softmax operation. These scores are then used to weight the transformed value features.

[0090] After determining the enhanced multimodal feature f m After that, the multimodal features are reshaped (i.e., Reshape in Formula 8) and upsampled (i.e., Upsample in Formula 8), and the upsampled results are Low-level image features obtained through skip connections with the image encoder The concatenation is performed along the channel dimension (i.e., [,] in Formula 9), ω represents the scale factor, and then the convolution layer is processed (i.e., Conv in Formula 9) and the activation function (i.e., ReL activation function in Formula 9) is activated to generate the final multimodal features. The formula is:

[0091]

[0092]

[0093] After the final multimodal features are determined, the second stage, the multi-expert uncertainty optimization stage, is entered. For a description of this part, please refer to Figure 1 Provide explanation.

[0094] In the second stage, the Multi-Expert Uncertainty Optimization (MEUR) component uses advanced methods such as Choquet integral (CI) to evaluate the uncertainty of the multi-expert model output. MEUR improves the reliability of segmentation by capturing subtle changes and consistency between expert predictions. The CI method specifically highlights the differences between expert outputs, providing a strong quantification of segmentation uncertainty. This refinement process not only improves segmentation accuracy, but also facilitates detailed evaluation of model outputs.

[0095] The multi-expert uncertainty optimization component includes an initial segmentation module, an uncertainty estimation module, and a segmentation optimization module.

[0096] An initial segmentation module is used to generate an initial segmentation result of an image using a multi-expert mixture model;

[0097] Uncertainty estimation module, used to generate expert uncertainty estimates;

[0098] The segmentation optimization module is used to optimize the initial segmentation results based on the expert uncertainty estimation.

[0099] The Multi-Expert Mixture (MoE) model is a neural network structure that weights and combines the outputs of multiple expert models (sub-models). The key idea is to introduce a gating network to dynamically select the contribution of each expert model, thereby improving the overall performance of the model.

[0100] Specifically, the initial segmentation module uses a multi-expert mixture model to generate the initial segmentation results of the image, including:

[0101] After receiving the input multimodal features f o After that, input x=f o , each expert model E i Generate a first output e i , that is, e i =E i (x);

[0102] Each first output is weighted with the gating network, and the weight of the gating network is g i =G i (x), and obtain the second output e corresponding to each expert model i ',Right now

[0103] The second output and the learnable mixture weight Φ i After weighting, we get the mixed output p of the multi-expert mixed model. That is the initial segmentation result.

[0104] Where M represents the number of expert models.

[0105] The learnable weight g i satisfy: And g i ≥0.

[0106] Subsequently, in order to further improve the performance of the model, the present invention uses a further uncertainty refinement mechanism to improve the MoE and enhance its expressiveness.

[0107] Choquet integral is a widely used method for fuzzy measurement, which has wide applications in decision theory, image processing and fuzzy systems. By combining the significance of input features and their weights, Choquet integral provides a comprehensive evaluation of multiple features. This method facilitates the integration of fuzzy sets and more accurately captures the interactions and dependencies between variables. Choquet integral is essential for combining feature weights and values, effectively managing uncertainty and ambiguity

[0108] In some embodiments, the uncertainty estimation module generates the expert uncertainty estimate specifically including:

[0109] According to the second output of each expert model and the mixed output of the multi-expert mixed model (for example, the second output and the mixed output of all expert models can be spliced ​​into a feature set), the Choquet fuzzy integral calculation is performed to obtain the mask m. The mask matrix is ​​expressed as follows: Among them, f σ(i) represents the eigenvalue of the i-th expert model after sorting the eigenvalues ​​of the expert models in ascending order, and σ(i) represents the index after sorting the eigenvalues ​​in descending order; represents the cumulative weight of the expert model after sorting from 0 to i, w (j) is the learnable fuzzy measure value of the expert model;

[0110] According to the initial segmentation results, determine the area A of the mask region in the image mask and the area of ​​the bounding box region A bbox ;

[0111] According to the area of ​​the mask region and the area of ​​the bounding box region, a threshold θ is calculated. For example, θ may be any suitable value. In some embodiments, the value may be

[0112] Using the threshold θ, the mask is converted into a binary mask, that is, the pixel feature m corresponding to each pixel in the mask is converted into i Compared with the threshold θ, if m i If the pixel feature is not less than the threshold θ, the pixel feature corresponding to the pixel is set to 1.0; otherwise, the pixel feature corresponding to the pixel is set to 0, thereby obtaining the expert uncertainty estimate. The binary mask is expressed as:

[0113]

[0114] Furthermore, the segmentation optimization module optimizes the initial segmentation results according to the expert uncertainty estimation, specifically including:

[0115] According to the binary mask, calculate the foreground mean p f and the background average p b , the specific formula can be expressed as:

[0116]

[0117]

[0118] Formula (11) represents the sum of the positions where the pixel is greater than or equal to 0.5 and then divided by the sum of the positions where the pixel is greater than or equal to 0.5 and rounded up to 1 (that is, the denominator represents the total number of pixels where the pixel is greater than or equal to 0.5), which represents the foreground average value. The background average value in formula (12) is similar. Among them, N represents the number of pixels; p i Represents the pixel value of the i-th pixel in the mixed output p of the multi-expert mixture model.

[0119] Next, the refinement parameter λ is set, and the refined prediction mask p is determined based on the refinement parameter λ, the binary mask, and the mixed output of the multi-expert mixture model. r , expressed as: p r =p+λ·m;

[0120] According to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined to achieve segmentation optimization.

[0121] In some embodiments, the final prediction mask is determined according to the relationship between the foreground average value, the background average value and the mixed output, specifically including:

[0122] If the mixed output is greater than the background average and the mixed output is less than the foreground average, the refined prediction mask is used as the final prediction mask;

[0123] Otherwise, the mixed output is used as the final prediction mask, that is, it can be expressed as:

[0124]

[0125] Among them, (p r >p b )∧(p r <p f ) indicates p r >p bEstablished and p r <p f Established.

[0126] The proposed CLV-SSIF is a new language-guided medical image segmentation framework. By integrating knowledge in the language and vision fields, CLV-SSIF enhances image analysis capabilities. MIGD promotes the interaction between the Text-SSM and Image-SSM modules, and uses cross-attention to extract contextual information, thereby improving segmentation accuracy. In addition, the MEUR component uses the CI method to quantify segmentation uncertainty, thereby improving the reliability and details of the results. Comparison results on two different datasets consistently show that the proposed method performs well in achieving high-precision segmentation. This progress provides a promising direction for future research and applications in medical imaging.

[0127] The advantages of the present invention are described below through specific experiments.

[0128] The experiments in this paper are implemented in the PyTorch framework. The training is performed on a high-performance device equipped with an NVIDIA GeForce RTX3090 GPU, which effectively manages the computing requirements. The initial learning rate is set to 3×10 -4 ,

[0129] A cosine annealing strategy is used as the learning rate scheduler to improve convergence. The AdamW optimizer is used to dynamically adjust the learning rate to enhance the optimization effect. The default batch size is 32, and all input images are resized to 224×224 pixels to ensure the consistency of the dataset. The number of experts is set to 4. The experiments are conducted on two medical datasets: the QaTa-COV19 dataset and the MosMedData+ dataset. Two standard metrics are used for evaluation: the Dice score and the mean intersection over union (mIoU), which are commonly used to evaluate the accuracy in object contour tasks. The model is trained for 200 epochs on the QaTa-COV19 dataset and 400 epochs on the MosMedData+ dataset to enhance the robustness and comprehensive learning of the model.

[0130] In order to study the performance of various current methods and our proposed CLV-SSIF method in medical image segmentation, we performed benchmark tests using different datasets. Table 1 shows the medical image segmentation experiments conducted on the QaTa-COV19 and MosMedData+ datasets, using CNN-based methods (such as U-Ne, U-Net++ and AttUNet) and hybrid methods (such as TransUNet and Swin Unet). In addition, this experiment also compares the latest multimodal segmentation models, including LViT, RecLMIS, LanGuideMedSeg and the proposed CLV-SSIF.

[0131] Table 1 Results of image segmentation using different methods

[0132]

[0133]

[0134] It should be noted that "×" in Table 1 means that text is not included; "√" means that text is included. The experimental results in Table 1 show that CLV-SSIF outperforms these state-of-the-art methods. Specifically, CLV-SSIF achieves Dice coefficients of 90.80% and 79.46%, and mIoU scores of 83.15% and 65.92% on the QaTa-COV19 and MosMedData+ datasets, respectively. These results are significantly improved over traditional CNN architectures and other hybrid models, demonstrating the effectiveness of integrating multimodal inputs. It is worth noting that CLV-SSIF maintains high accuracy and achieves the lowest FLOPs (11.1) with a moderate number of parameters (60 million), showing its computational efficiency.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An image segmentation method based on multimodal state space integration of uncertainty estimation, characterized in that: include: Acquire image data and text data corresponding to the image data; The CLV-SSIF model is used to segment images based on image data and text data. The CLV-SSIF model includes a multimodal spatial integration component and a multi-expert uncertainty optimization component. The multimodal spatial integration component includes an image encoder, a text encoder, and a multimodal interactive guided decoder. An image encoder, for receiving image data and generating image features; A text encoder, used to receive text features and generate text features; Multimodal interactive guided decoder, used to integrate image features and text features to generate multimodal features; The multi-expert uncertainty optimization component includes an initial segmentation module, an uncertainty estimation module, and a segmentation optimization module. An initial segmentation module is used to generate an initial segmentation result of an image using a multi-expert mixture model; Uncertainty estimation module, used to generate expert uncertainty estimates; The segmentation optimization module is used to optimize the initial segmentation results based on the expert uncertainty estimation.

2. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 1, characterized in that: The multimodal spatial integration component also includes adapters for fine-tuning text features to fit the outputs of image encoders at different stages.

3. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 1, characterized in that: The multimodal interactive guided decoder includes an image SSM module, a text SSM module, and a multimodal interaction module. The multimodal interactive guided decoder integrates image features and text features to generate multimodal features, including: Using the image SSM module, the image features are scanned through the state space to generate enhanced image features; Using the text SSM module, the text features are scanned in state space to generate enhanced text features; By utilizing the multimodal interaction module and adopting the multi-head cross-attention mechanism, the enhanced image features plus the image position encoding are used as the query, the enhanced text features plus the text position encoding are used as the key, and the enhanced text features are used as the value. Multi-head cross-attention calculation is performed to obtain the initial multimodal features.

4. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 3 is characterized in that: The multimodal interaction module also obtains enhanced multimodal features by residually connecting the initial multimodal features with the enhanced image features.

5. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 4 is characterized in that: The method also includes: after determining the enhanced multimodal features, reshaping and upsampling the multimodal features, and splicing the upsampling results with the low-level image features obtained by the image encoder through jump connections along the channel dimension, and then processing through convolution layers and activating activation functions to generate the final multimodal features.

6. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 1, characterized in that: Multiple expert mixture models and gating networks. The initial segmentation module uses multiple expert mixture models to generate the initial segmentation results of the image, including: After receiving the multimodal features, each expert model generates a first output; Each first output is weighted with the gating network to obtain a second output corresponding to each expert model; The second output is weighted with the learnable mixing weight to obtain the mixed output of the multi-expert mixture model, i.e., the initial segmentation result.

7. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 6, characterized in that: The uncertainty estimation module generates expert uncertainty estimates, including: According to the second output of each expert model and the mixed output of the multi-expert mixed model, Choquet fuzzy integral calculation is performed to obtain a mask; According to the initial segmentation result, determine the area of ​​the mask region and the area of ​​the bounding box region in the image; Calculate the threshold based on the area of ​​the mask region and the area of ​​the bounding box region; Using a threshold, the mask is converted into a binary mask to obtain an expert uncertainty estimate.

8. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 7, characterized in that: The segmentation optimization module optimizes the initial segmentation results based on the expert uncertainty estimate, including: Based on the binary mask, calculate the foreground average and background average; Setting refinement parameters, and determining a refined prediction mask according to the refinement parameters, the binary mask, and the mixed output of the multi-expert mixture model; According to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined to achieve segmentation optimization.

9. The image segmentation method based on multimodal state space integration of uncertainty estimation according to claim 8, characterized in that: According to the relationship between the foreground average value, the background average value and the mixed output, the final prediction mask is determined, including: If the mixed output is greater than the background average and the mixed output is less than the foreground average, the refined prediction mask is used as the final prediction mask; Otherwise, the mixed output is used as the final prediction mask.

Images