Multi-modal medical image segmentation method and device, computer device and storage medium

Abstract

The application discloses a kind of multi-modal medical image segmentation method, device, computer equipment and storage medium, comprising: obtaining the multi-modal medical image of the same examinee and pre-processing and spatial registration;The target segmentation concept is standardized and analyzed to generate prompt;Using basic segmentation model obtains multiple candidate segmentation results of each modality;Constitute credibility gating model, assess the credibility score of each candidate result and carry out screening, obtain the credible instance of each modality;Establish the corresponding relationship between cross-modal credible instance to form instance group;Each instance group is executed evidence weighted fusion and conflict resolution, and the final segmentation result of each target is obtained;Output multi-instance segmentation result and corresponding instance-level uncertainty index.The application can effectively reduce the false detection and result fluctuation of basic model in multi-modal image, improve the consistency and reliability of cross-modal segmentation, and provide robust segmentation result for clinical screening.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a multimodal medical image segmentation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Medical image segmentation is a key technology for computer-aided diagnosis, treatment planning, and postoperative evaluation. Its goal is to automatically and accurately extract organs, tissues, or lesion regions of interest from images such as CT, MRI, PET, and ultrasound. In recent years, with the diversification of imaging technologies, the fusion of multiple modalities in imaging for comprehensive diagnosis of the same subject has become a clinical routine. This helps to improve the comprehensiveness and accuracy of diagnosis by utilizing complementary information from different modalities. Against this backdrop, how to achieve robust and accurate multimodal medical image segmentation has become a research hotspot and challenge in the field of medical image analysis. Traditional segmentation methods typically rely on specific modal designs or require large amounts of targeted labeled data for model training. When faced with diverse modal combinations and significantly different imaging characteristics, their generalization ability and stability are often limited. In recent years, the emergence of large-scale visual basic models, especially cue-enabled segmentation models, has brought a new paradigm to general image segmentation. These models can achieve zero-sample or small-sample segmentation of any target based on diverse cues such as user-input text, points, and bounding boxes, demonstrating strong generalization ability.

[0003] In the process of developing this invention, the inventors realized that the existing technology has at least the following technical problems: When existing suggestible segmentation models are directly applied to complex and varied medical images, especially in multimodal scenarios, medical images often suffer from quality issues such as noise, artifacts, and low contrast. The model is easily interfered with and generates false positive segmentation candidates (false detections). Furthermore, the output results fluctuate significantly between different slices or modalities, resulting in poor repeatability. Secondly, due to the different imaging principles and characteristics of each modality, the morphology, boundary clarity, and visibility of the same anatomical structure vary greatly in different images, leading to inconsistent segmentation results generated by the model across modalities. Simple fusion strategies (such as averaging or voting) are prone to boundary blurring, positioning drift, or missed detections.

[0004] In summary, current technologies for multimodal medical image segmentation tasks have not yet effectively addressed a series of interconnected technical problems, including high false positive rates, unstable outputs, inconsistent cross-modal results, and a lack of instance-level interpretability evaluation metrics. These problems stem from differences in image quality, conflicts in modal characteristics, and uncertainties in the underlying model. Therefore, there is an urgent need for a new technical solution that can leverage the powerful capabilities of general segmentation models while effectively suppressing their unreliable outputs, intelligently fusing multimodal evidence, and ultimately providing stable, reliable, and credible segmentation results. Summary of the Invention

[0005] This invention provides a multimodal medical image segmentation method, apparatus, computer equipment, and storage medium to improve the accuracy and reliability of modal medical image segmentation.

[0006] To address the aforementioned technical problems, embodiments of this application provide a multimodal medical image segmentation method, including:

[0007] Acquire multimodal medical images of the same subject, preprocess and spatially register each modality of the images to obtain a multimodal image set registered to the same reference space;

[0008] The input target segmentation concept is standardized and structured using medical terminology, and segmentation prompts are generated based on the parsing results.

[0009] The images of each modality in the multimodal image set and the segmentation prompt information are respectively input into the basic segmentation model to obtain multiple candidate segmentation results corresponding to each modality and the confidence information of each candidate segmentation result;

[0010] Based on the credibility gating model, the credibility of each candidate segmentation result is evaluated to obtain a credibility score for each candidate segmentation result. Then, based on the credibility score, a set of credible instances under each modality is obtained.

[0011] Calculate the matching cost between different modalities of trusted instances in the trusted instance set, and perform instance matching based on the matching cost to form multiple instance groups, each instance group corresponding to a real anatomical or pathological target;

[0012] The target segmentation result is obtained by weighted fusion based on the credibility score of each trusted instance in the group and the sensitivity weight of its modality; when the segmentation results of each instance in the group conflict more than a threshold, an adjudication strategy is executed.

[0013] The output consists of a multi-instance segmentation result composed of the fused segmentation results of all targets.

[0014] Optionally, the credibility evaluation of each candidate segmentation result based on the credibility gating model, to obtain a credibility score for each candidate segmentation result, includes:

[0015] For each candidate segmentation result, the image semantic features and mask shape features of the candidate segmentation result are extracted, and a conditional vector containing information on concept existence, image quality and modality sensitivity is constructed.

[0016] The image semantic features, mask shape features, and conditional vector are input into the credibility gating model to calculate the credibility score of the candidate segmentation result;

[0017] The credibility score is compared with a preset threshold to filter out the set of credible instances.

[0018] Optionally, the extraction of image semantic features and mask shape features of the candidate segmentation results includes:

[0019] The image encoder of the basic segmentation model is used to obtain the deep embedding feature map of the input image;

[0020] The image semantic features are obtained by performing region pooling on the deep embedding feature map using a binary mask of the candidate segmentation results.

[0021] Optionally, the step of inputting the image semantic features, mask shape features, and conditional vector into the confidence gating model to calculate the confidence score of the candidate segmentation result includes:

[0022] The image semantic features, mask shape features, and conditional vector are input into a conditional variational autoencoder;

[0023] Obtain the reconstruction error and posterior uncertainty of the latent variables from the output of the conditional variational autoencoder;

[0024] The confidence score is calculated using a parameterized fusion function based on the reconstruction error, the posterior uncertainty, and the conditional vector.

[0025] Optionally, the matching cost between different modalities of trusted instances in the set of trusted instances includes:

[0026] Calculate the mask space similarity between any two trustworthy instances from different modalities;

[0027] Calculate the feature space distance between the image semantic features of the two trusted instances;

[0028] The matching cost is obtained by weighted summation of the mask space similarity and the feature space distance.

[0029] Optionally, the weighted fusion based on the credibility scores of each trusted instance within the group and the sensitivity weights of its respective modality, to obtain the fusion segmentation result of the target, includes:

[0030] Assign a fusion weight to each trusted instance within the instance group, the fusion weight being the product of the instance's trust score and the sensitivity weight of its modality;

[0031] Using the fusion weights, the segmentation probability maps of each trusted instance are weighted and averaged to obtain the fusion probability map of the target.

[0032] The fusion probability map is binarized to obtain the fusion segmentation result.

[0033] Optionally, the adjudication strategy is as follows: when a conflict occurs, during the weighted fusion process, only the segmentation probability map corresponding to the credible instance with the largest fusion weight is adopted as the primary evidence for generating the fusion probability map.

[0034] To address the aforementioned technical problems, this application also provides a multimodal medical image segmentation device, comprising:

[0035] The image registration module is used to acquire multimodal medical images of the same subject, preprocess and spatially register each modality of image to obtain a multimodal image set registered to the same reference space;

[0036] The text parsing module is used to perform medical terminology standardization and structured parsing on the input target segmentation concept, and generate segmentation prompt information based on the parsing results;

[0037] The information segmentation module is used to input the images of each modality in the multimodal image set and the segmentation prompt information into the basic segmentation model respectively, and obtain multiple candidate segmentation results corresponding to each modality and the confidence information of each candidate segmentation result;

[0038] The credibility assessment module is used to assess the credibility of each candidate segmentation result based on the credibility gating model, obtain a credibility score for each candidate segmentation result, and filter according to the credibility score to obtain a set of credible instances under each modality.

[0039] The instance matching module is used to calculate the matching cost between trusted instances of different modalities in the trusted instance set, and to perform instance matching based on the matching cost to form multiple instance groups, each instance group corresponding to a real anatomical or pathological target;

[0040] The weighted fusion module is used to perform weighted fusion based on the credibility score of each trusted instance in the group and the sensitivity weight of its modality to obtain the fused segmentation result of the target; when the segmentation results of the instances in the group conflict more than a threshold, an adjudication strategy is executed.

[0041] The results output module is used to output a multi-instance segmentation result consisting of the fused segmentation results of all targets.

[0042] To address the aforementioned technical problems, this application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned multimodal medical image segmentation method.

[0043] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned multimodal medical image segmentation method.

[0044] The multimodal medical image segmentation method, apparatus, computer equipment, and storage medium provided in this invention acquire multimodal images and complete spatial registration, providing a consistent coordinate basis for subsequent cross-modal analysis; by standardizing and generating prompts for target segmentation concepts, the model's understanding and accuracy in pointing to complex medical terms are improved; a credibility-gated model scores and filters candidate results for each modality, effectively eliminating low-quality and unreliable segmentation proposals; by calculating cross-modal instance matching costs and establishing correspondences, segmentation results of the same real target in different modalities are reliably associated; further, weighted fusion is performed based on credibility and modality weights, and an adjudication strategy is activated when conflicts exceed the threshold, effectively integrating multi-source information and resolving inconsistencies between modalities. This improves the accuracy and consistency of multimodal medical image segmentation. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0047] Figure 2 This is a flowchart of an embodiment of the multimodal medical image segmentation method of this application;

[0048] Figure 3 This is a reference example diagram of a multimodal data source for this application;

[0049] Figure 4 This is a schematic diagram of the structure of one embodiment of the multimodal medical image segmentation device according to this application;

[0050] Figure 5 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0052] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figure 1 ,like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0055] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0056] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0057] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0058] It should be noted that the multimodal medical image segmentation method provided in this application embodiment is executed by a server, and correspondingly, the multimodal medical image segmentation device is set in the server.

[0059] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. The terminal devices 101, 102, and 103 in this embodiment can specifically correspond to application systems in actual production.

[0060] Please see Figure 2 , Figure 2 This invention illustrates a multimodal medical image segmentation method provided by an embodiment of the present invention, which is applied to... Figure 1 Taking the server-side as an example, the details are as follows:

[0061] S201: Acquire multimodal medical images of the same subject, preprocess and spatially register each modality of the images to obtain a multimodal image set registered to the same reference space.

[0062] In medical diagnosis, the anatomical or functional information provided by a single imaging modality is often limited. To obtain more comprehensive diagnostic information, multiple imaging techniques are often used in combination for the same subject in clinical practice. These include computed tomography (CT) and magnetic resonance imaging (MRI) that provide high-resolution anatomical structures, positron emission tomography (PET) that reflects metabolic function, and real-time dynamic ultrasound (US) and endoscopic images. This embodiment first acquires such a collection of multimodal medical images of the same subject, forming the raw data basis for subsequent analysis.

[0063] Due to differences in the physical principles, scanning protocols, and patient positioning of various imaging devices, the directly acquired images of different modalities do not exhibit consistency in pixel intensity distribution, spatial resolution, and geometric coordinates. To enable subsequent processing to compare and fuse multi-source information within a unified framework, a series of standardization and alignment operations must be performed on the original images. This process mainly includes two core steps: preprocessing and spatial registration.

[0064] Preprocessing aims to improve the quality and comparability of single-modal images. Specific operations include, but are not limited to: using filters (such as Gaussian filtering and median filtering) to denoise the images and suppress random noise; performing resampling to adjust each image to a consistent voxel spacing; performing intensity normalization to eliminate signal intensity differences caused by different scanning devices or protocols, for example, using Z-score normalization, which standardizes the intensity based on the mean and standard deviation of each modality image; and necessary cropping and artifact suppression operations. After preprocessing, the image has a more consistent internal intensity distribution, which is beneficial for feature extraction by the model.

[0065] The goal of spatial registration is to geometrically align the images of all modalities to a common reference coordinate system. In this embodiment, one modality (such as high-resolution CT) is selected as the reference space, and for each of the other modalities, a spatial transformation model is calculated using an image registration algorithm. This transformation model can be a rigid transformation describing overall displacement and rotation, an affine transformation allowing scaling and shearing, or a non-rigid deformation field capable of modeling local deformations. Specifically, the registration algorithm includes, but is not limited to, optimization methods based on similarity metrics such as mutual information and normalized cross-correlation. By applying the calculated transformation, the images of all other modalities are mapped to the reference space, thereby generating a spatially aligned multimodal image set.

[0066] It should be noted that the multimodal image dataset in this embodiment includes both 2D and 3D image data, and adopts a strategy of using two-dimensional data as a unified processing unit. The model structure is designed based on two-dimensional medical images. For three-dimensional volume data, it is first sliced ​​along the slice dimension, representing it as a series of two-dimensional slices, and then input into the model slice by slice to complete training and inference. During the training phase, samples can be constructed by randomly selecting slices from the volume data, thereby maintaining the data form consistent with the two-dimensional data. During the inference phase, the slice-by-slice method is also adopted, and finally the results of each slice are recombined to form a complete volume data segmentation output.

[0067] Figure 3 This is an example diagram of a multimodal image source in this application. In this example, the multiple modalities include CT, MRI, PET, ultrasound, and endoscopic images. Let the multimodal medical images of the same subject be... ,in This represents the modality index, where each modality image is defined in its coordinate space. .

[0068] Furthermore, for each mode Preprocessing operations are performed, including denoising, resampling, intensity normalization, cropping, and artifact suppression. The normalization operation uses Z-score normalization, as shown in formula (1), to obtain the preprocessed modes. :

[0069] (1)

[0070] in, and Modal The mean and standard deviation of the intensity.

[0071] Furthermore, a reference mode is selected. Find the transformation from each mode to the reference space. The transformation includes one of rigid, affine, or deformation registration. As shown in formula (2), the modal... The image can be mapped to the reference space to obtain:

[0072] (2)

[0073] in For reference space.

[0074] S202: Perform medical terminology standardization and structured parsing on the input target segmentation concept, and generate segmentation prompt information based on the parsing results.

[0075] The input for medical image segmentation tasks typically originates from the diagnostic needs of clinicians, and its expression may be a brief free text description or a structured field, such as "liver," "right upper lobe nodule," or "pancreatic tumor." However, natural language expressions suffer from problems such as synonyms, abbreviations, ambiguous hierarchical relationships, and potential ambiguities, and directly inputting them into the model may lead to misunderstandings. Therefore, this embodiment first performs medical terminology standardization on the input target segmentation concept.

[0076] This process maps free text or non-standard expressions to a unified standard medical terminology system, such as the Systematic Medical Terminology Set (SNOMED CT) or the Unified Medical Language System (UMLS). The mapping process typically combines medical knowledge graphs with natural language processing techniques, and specific implementations include, but are not limited to: dictionary mapping based on string matching and edit distance, or semantic disambiguation using pre-trained biomedical language models. During this process, the system simultaneously performs synonym merging, negation word detection and processing, and constraint verification based on anatomical hierarchical relationships, ultimately outputting an unambiguous, standardized conceptual expression.

[0077] To further refine the segmentation model, this embodiment performs structured parsing on the normalized concepts, decomposing them into a combination of atomic concepts. These atomic concepts aim to represent the target from different dimensions, including but not limited to: organ concepts indicating major anatomical structures, location concepts describing more precise positions, lateral concepts distinguishing left and right sides, pathological category concepts defining the nature of lesions, perspective concepts identifying imaging viewpoints, and modifying concepts expressing attributes such as size and density.

[0078] Through this structured decomposition, even if the input conceptual information is incomplete, the system can retain blanks or use default values ​​in the corresponding dimensions, thereby forming a flexible and expressive intermediate representation.

[0079] Based on the aforementioned structured set of atomic concepts, this embodiment generates segmentation hints that can be understood by a basic segmentation model (such as SAM3). These hints typically consist of text hints and optional example hints. Text hints can be directly generated from atomic concepts into natural language phrases. Example hints provide spatial guidance; they can take the form of sparse hints such as points or boxes provided by user interaction, or, in some automated processes, possible regions initially inferred by the model based on text hints. Finally, a composite hint combining text and examples is constructed and passed to the downstream segmentation model. This transforms unstructured clinical descriptions into standardized instructions that are accurately parsed by machines and rich in semantic and spatial cues, significantly improving the model's accuracy in understanding and executing complex, professional medical segmentation intentions, and providing objective data support for obtaining high-quality candidate segmentation results.

[0080] In a specific example, the input target segmentation concept This can be set as a free text or structured field for liver, right upper lobe lung nodules, or pancreatic tumors. The concepts are standardized by mapping them to the standard medical terminology system. It also performs synonym merging, ambiguity resolution, hierarchical constraints, and negation word processing.

[0081] Furthermore, for the standardization concept The atomic concept, which includes organs, parts, target categories, and attributes, is represented by formula (3):

[0082] (3)

[0083] Where A is the set of atomic concepts. Indicates an organ. Indicates location. Indicates the side, Indicates the type of lesion. Indicates perspective, This indicates an attribute modifier. When a certain type of information is missing, the corresponding atomic concept can be empty.

[0084] Furthermore, based on the set of atomic concepts Generate noun phrase suggestions ;This can then be used to generate further example hints. For example, a box or point cue can be given to indicate the specific region of a target instance to the model. The final input SAM3 concept cue combination is as follows: .

[0085] S203: Input the images and segmentation prompts of each modality in the multimodal image set into the basic segmentation model respectively, and obtain multiple candidate segmentation results corresponding to each modality and the confidence information of each candidate segmentation result.

[0086] To fully leverage the powerful zero-shot and cue-learning capabilities of general segmentation models, this embodiment selects Segment Anything Model (SAM) and its evolved versions (such as SAM3) as the base segmentation model. This model falls under the category of cue-enabled segmentation models and can output corresponding segmentation masks based on the image and accompanying cue information (such as text, points, and bounding boxes).

[0087] In step S203, each modal image in the multimodal image set that has been registered to the same reference space obtained in the previous steps is paired with the generated segmentation prompt information and input into the basic segmentation model for independent inference.

[0088] For each modality input, the base segmentation model outputs a set of candidate segmentation instances. Each candidate instance contains a soft mask, where each pixel value represents the probability that the pixel belongs to the target object, ranging from 0 to 1. Simultaneously, the model outputs an instance localization score for each candidate instance, quantifying the spatial matching degree between the candidate region and the target indicated by the prompt. Furthermore, the model outputs a concept existence probability through its internal mechanisms (such as a global existence token), representing the likelihood that the concept described by the input segmentation prompt exists in the entire current modality image. For 3D volume data, this embodiment employs a slice-by-slice inference strategy, processing each 2D slice of the 3D volume individually to obtain slice-level candidate results, which are then aggregated along the slice direction to reconstruct 3D candidate segmentation instances.

[0089] In step S203, each modal image independently generates a set of candidate segmentation results with preliminary confidence assessments (including instance localization scores and concept existence probabilities). These candidate results reflect the initial identification and localization of the target by the base model based on single-modal information and general prior knowledge. The output is diverse and may include true positive targets, false positive proposals due to image noise or modal characteristics, and repeated or partial segmentation of the same target. This efficiently utilizes the generalization ability of the cutting-edge base model and provides rich, original segmentation proposals from various modalities with initial quality assessments for subsequent processing. These proposals constitute the direct input for subsequent fine-grained confidence assessments and cross-modal fusion.

[0090] In a specific example, for each mode ,Will Input prompts Input a SAM3 model, and SAM3 outputs a set of soft masks for candidate instances. and corresponding instance location score ,in For the first A soft mask for each candidate instance. It is the first The number of candidates output in each modality. When the image is 2D, a 2D candidate mask is directly obtained. When the image is 3D, inference is obtained from slices of volume data. Then, they are aggregated into three dimensions to obtain a three-dimensional candidate mask.

[0091] Furthermore, SAM3 outputs the existence probability of a concept through a globally existing token. , is used to indicate whether the noun phrase concept exists in the input image. The total candidate score used in inference is expressed by formula (4):

[0092] (4)

[0093] The soft mask is thresholded using formula (5) to obtain a binary candidate mask:

[0094] (5)

[0095] in, For mask threshold, This is an indicator function.

[0096] S204: Based on the credibility gating model, the credibility of each candidate segmentation result is evaluated to obtain the credibility score of each candidate segmentation result. Then, the credibility scores are used to filter and obtain the set of credible instances under each modality.

[0097] In one optional implementation, based on a credibility gating model, the credibility of each candidate segmentation result is evaluated to obtain a credibility score for each candidate segmentation result, including:

[0098] For each candidate segmentation result, extract the image semantic features and mask shape features of the candidate segmentation result, and construct a conditional vector containing information on concept existence, image quality and modality sensitivity;

[0099] Image semantic features, mask shape features, and conditional vectors are input into a confidence gating model to calculate the confidence score of the candidate segmentation result.

[0100] The credibility score is compared with a preset threshold to filter out a set of credible instances.

[0101] In an optional implementation, extracting the image semantic features and mask shape features of the candidate segmentation results includes:

[0102] The image encoder of the basic segmentation model is used to obtain the deep embedding feature map of the input image;

[0103] The binary mask of the candidate segmentation result is used to perform region pooling on the deep embedding feature map to obtain the semantic features of the image.

[0104] In an optional implementation, the image semantic features, mask shape features, and conditional vectors are input into a confidence gating model to calculate the confidence score of the candidate segmentation result, including:

[0105] The image semantic features, mask shape features and conditional vectors are input into a conditional variational autoencoder;

[0106] Obtain the reconstruction error and latent variable posterior uncertainty of the conditional variational autoencoder output;

[0107] The confidence score is calculated using a parameterized fusion function based on reconstruction error, posterior uncertainty, and conditional vector.

[0108] While the candidate segmentation results generated by the basic segmentation model have an initial confidence level, their evaluation is mainly based on the model's forward propagation score, failing to fully consider the unique complexities of medical images, such as image quality fluctuations, differences in sensitivity between modalities, and the rationality of feature distribution. Therefore, this embodiment introduces a dedicated confidence gating model to perform a secondary, more in-depth confidence evaluation on each candidate segmentation result from each modality. This step aims to identify truly reliable segmentation instances from a large number of candidate proposals, effectively suppressing false positives and output fluctuations.

[0109] Credibility assessment begins with multi-dimensional feature extraction and condition construction of candidate results. First, the powerful image encoder of the base segmentation model is used to obtain a deep embedding feature map of the input image, which contains rich semantic information. For each candidate segmentation result, its binarized mask is used to extract the semantic features of the corresponding region from the deep embedding feature map.

[0110] Specifically, features within the mask region are aggregated through pooling operations, such as global average pooling, to obtain a fixed-dimensional semantic feature vector. This vector summarizes the visual appearance and semantic context of the target region. Simultaneously, a set of statistical and shape features, such as area, perimeter, compactness, and Hu moments, can be extracted from the candidate mask itself to form a shape feature vector. These two features are then concatenated to form a joint feature describing the candidate.

[0111] On the other hand, a conditional vector reflecting the evaluation context is constructed. This vector integrates three key types of information: the concept existence probability provided by the base model, indicating the prior likelihood of the target appearing in the overall image; an image quality score calculated from the current modality image through a learnable quality assessment network, which can be implemented based on an encoder-regressor architecture; and a modality sensitivity weight reflecting the inherent fit between the current imaging modality and the target concept to be segmented, which can be learned from the data or preset based on prior knowledge. This conditional vector provides an important auxiliary basis for credibility assessment.

[0112] In one specific implementation, the credibility gating model employs a conditional variational autoencoder architecture. During training, this model learns the typical distribution of credible segmentation instances in the feature space. During evaluation, candidate joint features and conditional vectors are input into the encoder. The encoder outputs a posterior distribution of a latent variable, typically modeled as a Gaussian distribution, thus obtaining the mean and variance of the distribution. The decoder then attempts to reconstruct the input features based on the latent variables sampled from this distribution and the conditional vectors.

[0113] The model outputs two key signals: first, the reconstruction error, which is the difference between the original joint features and the reconstructed features, reflecting the degree to which the current candidate features deviate from the credible distribution; and second, the mean variance of the posterior distribution of the latent variables, which is the posterior uncertainty, reflecting the fuzziness or confidence level of the model's encoding of the candidate features.

[0114] Finally, the credibility score of the candidate segmentation result is calculated by combining the reconstruction error, posterior uncertainty, and conditional vector through a parameterized fusion function. This function can be implemented using a linear combination with learnable weights and a sigmoid function. The calculated credibility score is a scalar value between 0 and 1, with a higher score indicating a more credible candidate result. By comparing the credibility score of each candidate with a preset threshold, and possibly combining it with its initial total score for double filtering, high-credibility instances can be filtered from the candidate sets of each modality, forming independent sets of credible instances for each modality.

[0115] In a specific example, a conditional variational autoencoder confidence-gated model is constructed to perform a secondary confidence evaluation on candidate instances of SAM3. Reconstruction error and posterior uncertainty are used to suppress false detections and fluctuations. Specifically, this includes:

[0116] Step 1: Construct conditional variables for concept existence, image quality, and modal sensitivity, respectively. Use the variables obtained in Step 8. As a condition for the existence of a concept; calculate a quality score for each modality. As an image quality condition, it is calculated using formula (6):

[0117] (6)

[0118] in, For quality feature extraction function, It's the Sigmoid function. (Based on the concept set) With mode Adaptive learning yields sensitivity weights. The final condition vector is defined as shown in formula (7):

[0119] (7)

[0120] in, This is the final condition vector.

[0121] Step 2, construct candidate image-mask joint features. For each candidate mask... Image embedding features are obtained through the SAM3 image encoder. Use a mask Extracting the candidate instance region from the feature map yields a semantic summary of the candidate instance in the image feature space, such as whether it resembles a nodule and whether its edge features are reasonable. Mask region pooling features. The following formula (8) can be used for calculation:

[0122] (8)

[0123] in, It is a pooling operation. It is element-wise multiplication. It is a deep feature vector at the semantic level. The statistical and shape feature vectors are then concatenated with candidate joint features to form the input of the conditional variational autoencoder. .

[0124] Step 3: Construct a conditional variational autoencoder to allow the model to learn the state of the credible candidate in the feature space. The encoder outputs the posterior latent variable, which is expressed by formula (9):

[0125] (9)

[0126] in, It is the approximate posterior distribution of variational inference, and the parameter is , and These represent the encoder output mean and standard deviation, respectively. The decoder uses the obtained values... and conditions The features are reconstructed as shown in formula (10):

[0127] (10)

[0128] in, It's a decoder.

[0129] Step 4: Convert the two signals of the conditional variational autoencoder into a final gated score. The reconstruction error is used to reflect the degree of deviation of the features from the model; the higher the error, the lower the reliability, as shown in formula (11):

[0130] (11)

[0131] in, This represents the reconstruction error.

[0132] Posterior uncertainty reflects the dispersion of the latent variable distribution. Higher uncertainty indicates that the feature deviates further from the credible distribution, and the greater the credible risk. The mean of the posterior variance of the latent variables is used to characterize posterior uncertainty. As shown in formula (12):

[0133] (12)

[0134] in, For latent variable dimensions.

[0135] Furthermore, the candidate credibility is calculated comprehensively, as shown in formula (13), by fusing three factors: error suppression, uncertainty suppression, and conditional gain.

[0136] (13)

[0137] in, , , , These are learnable parameters.

[0138] Furthermore, credibility gating is performed. Candidate instances are gated based on their credibility scores to obtain a set of credible instances, as shown in formula (14):

[0139] (14)

[0140] in, As a credibility threshold, This is the threshold for the candidate total score.

[0141] This embodiment combines deep feature reconstruction consistency, distribution uncertainty, and multi-dimensional contextual conditions to achieve a refined and robust evaluation of the reliability of candidate segmentation results. This provides a rigorously controlled input for subsequent cross-modal fusion, significantly reducing the false detection rate and improving the stability of the system output.

[0142] S205: Calculate the matching cost between different modalities of trusted instances in the trusted instance set, and perform instance matching based on the matching cost to form multiple instance groups, each instance group corresponding to a real anatomical or pathological target.

[0143] In one optional implementation, calculating the matching cost between trusted instances of different modalities in the trusted instance set includes:

[0144] Calculate the mask space similarity between any two trustworthy instances from different modalities;

[0145] Calculate the feature space distance between the image semantic features of two trusted instances;

[0146] The matching cost is obtained by weighted summation of the mask space similarity and the feature space distance.

[0147] After confidence screening, each modality outputs a set of highly confident segmentation instances. However, these instances are generated independently and cross-modal associations have not yet been established. To integrate multimodal information to achieve collaborative analysis of the same real anatomical or pathological target, this embodiment needs to correctly match and classify these instances, which originate from different modalities but may describe the same target.

[0148] In this embodiment, the calculation of the matching cost takes into account both the similarity of instances in terms of spatial geometry and deep semantic features.

[0149] First, calculate the mask spatial similarity between any two reliable instances from different modalities. This is achieved by comparing their binary segmentation masks, with commonly used metrics including the Dice similarity coefficient and the Jaccard index (intersection over union ratio). Taking the Dice coefficient as an example, it calculates the ratio of twice the area of ​​the intersection of two masks to the sum of their areas, with a value range of [0,1]. A higher value indicates better spatial overlap, reflecting the consistency of their anatomical spatial locations.

[0150] Next, the distance between the two instances in the feature space is calculated. The features are the image semantic feature vectors extracted and aggregated through pooling in the credibility assessment step. By calculating metrics such as Euclidean distance and cosine distance between the two feature vectors, the similarity in visual semantic content can be assessed. For example, instances of liver tumors should have relatively similar semantic features, while the feature distance between instances of vascular structures should be larger.

[0151] Finally, the normalized spatial similarity (e.g., the 1-Dice coefficient can be converted into a difference) is linearly weighted and summed with the feature space distance to form a comprehensive matching cost. This cost function balances both spatial alignment and semantic content consistency. Based on the matching cost matrix calculated between all cross-modal trusted instance pairs, this embodiment uses graph matching or combinatorial optimization algorithms to find the globally optimal matching relationship. Specific implementations include, but are not limited to, the Hungarian Algorithm or the minimum cost maximum flow algorithm. These algorithms can effectively solve the one-to-one matching constraint problem, ensuring that an instance in one modality corresponds to at most one instance in another modality.

[0152] The Dice coefficient, also known as the Dice index, is a statistical metric used to measure the similarity between two sets or samples. It is widely used in image segmentation, medical image analysis, text matching, and other fields. Its core idea is to evaluate the consistency between predicted and actual results by calculating the relationship between their intersection and union.

[0153] In a specific example, the trusted instances of each modality after gating are aligned to the same real object, then evidence-weighted fusion is performed, and a decision is made in case of conflict, outputting the final segmentation and uncertainty. For any set of trusted instances of two modalities, the Dice index between instances is calculated using formula (15) to measure the degree of spatial overlap between the two masks:

[0154] (15)

[0155] in, For example With examples The Dice metric.

[0156] Furthermore, feature distance is introduced. The matching cost is constructed as shown in formula (16):

[0157] (16)

[0158] in, For example With examples The matching cost.

[0159] Cross-modal correspondence groups are obtained using the Hungarian algorithm or minimum cost matching. Each This represents the set of corresponding instances of the same lesion under different modalities.

[0160] Through the above process, all trusted instances are divided into several disjoint instance groups. Each group contains instances from one or more modalities, which are determined to depict the same real target. By fusing low-level spatial geometric information with high-level deep semantic information, a robust cross-modal instance association mechanism is constructed, which can effectively address matching ambiguities caused by differences in modal imaging and inconsistent segmentation boundaries.

[0161] S206: Based on the credibility score of each trusted instance in the group and the sensitivity weight of its modality, a weighted fusion is performed to obtain the fusion segmentation result of the target; when the segmentation results of each instance in the group conflict more than the threshold, an adjudication strategy is executed.

[0162] In one optional implementation, a weighted fusion is performed based on the credibility score of each trusted instance within the group and the sensitivity weight of its respective modality to obtain the fusion segmentation result of the target, including:

[0163] Assign a fusion weight to each trusted instance within the instance group. The fusion weight is the product of the instance's trust score and the sensitivity weight of its modality.

[0164] Using fusion weights, the segmentation probability maps of each trusted instance are weighted and averaged to obtain the fusion probability map of the target.

[0165] The fusion probability map is binarized to obtain the fusion segmentation result.

[0166] In one optional implementation, the adjudication strategy in conflict resolution is as follows: when a conflict occurs, during the weighted fusion process, only the segmentation probability map corresponding to the credible instance with the largest fusion weight is adopted as the main evidence for generating the fusion probability map.

[0167] After completing cross-modal instance matching and constructing instance groups, each group contains credible instances from one or more modalities corresponding to the same real target. To generate a more accurate and robust final segmentation result, this embodiment requires intelligent fusion of the segmentation evidence provided by each instance within a group. However, due to the inherent differences in imaging characteristics, resolution, and contrast of different modalities, there may be significant inconsistencies, i.e., conflicts, between the segmentation masks of each instance within a group. Therefore, this embodiment employs a strategy that combines evidence-weighted fusion with adaptive conflict resolution.

[0168] The core of evidence-weighted fusion lies in assigning a fusion weight to each credible instance, which comprehensively reflects the credibility of the instance itself and the value of its modality to the current segmentation task.

[0169] Specifically, for a trustworthy instance within a group, its fusion weight is calculated as the product of the trustworthiness score obtained by the instance in the previous step and the sensitivity weight of its modality. This calculation method allows instances that are assessed as highly trustworthy within a single modality and whose modality is determined to be sensitive to the target in a multimodal context (e.g., CT is sensitive to calcifications, MRI is sensitive to soft tissue tumors) to gain greater influence in the fusion process.

[0170] Subsequently, these fusion weights are used to perform a weighted average of the soft masks (i.e., segmentation probability maps) of all trusted instances within the group. The result of the weighted average is a probability map that integrates multi-source information, called the fusion probability map, where each pixel value represents the multimodal comprehensive probability that the pixel belongs to the target. Finally, by applying a preset threshold to binarize the fusion probability map, the final fusion segmentation result of the target, i.e., a deterministic binary mask, can be obtained.

[0171] While weighted fusion can integrate information, simple averaging can lead to ambiguous or erroneous fusion results when the evidence provided by different modalities is severely divergent. Therefore, this embodiment introduces a conflict detection and adjudication mechanism. The degree of conflict is assessed by quantifying the consistency between the segmentation masks of each instance within a group. Specifically, this can be achieved by calculating the average Dice similarity coefficient between all instance pairs and subtracting this average from 1 as the conflict level indicator.

[0172] When the calculated degree of conflict exceeds a preset threshold, it indicates that the differences between the modal results within the group are too large, and continuing to perform the conventional weighted average carries a high risk. In this case, this embodiment initiates a decision-making strategy: abandoning equal weighting of all evidence, instead adopting only the segmentation probability map corresponding to the credible instance with the largest fusion weight as the sole or primary evidence for generating the fusion probability map, while the contributions of other instances are downweighted or ignored. This strategy is equivalent to selecting the most reliable (high credibility) and most relevant (high modality sensitivity) single piece of evidence as the basis for decision-making when there are serious contradictions in multimodal evidence, thereby avoiding misleading results caused by forcibly fusing highly conflicting evidence.

[0173] In a specific example, group The soft masks of each mode are fused into a more reliable soft mask. Let the modes... The soft mask for the corresponding instance is As shown in formula (17), its fusion weight is determined by the product of modal sensitivity and credibility:

[0174] (17)

[0175] in, To integrate weights, For modal sensitivity, For credibility.

[0176] Furthermore, the fusion concept map is obtained through formula (18):

[0177] (18)

[0178] in, To prevent division by zero of extremely small constants.

[0179] Furthermore, the fused probability map is transformed into the final segmentation mask, and the binary fusion mask is output, expressed by formula (19):

[0180] (19)

[0181] in, This is the fusion threshold.

[0182] Considering the inconsistencies between cross-modal candidate masks, this example introduces a conflict degree to detect the degree of inconsistency within a group. The intra-group conflict degree is characterized by 1-mean Dice, as shown in Equation (20):

[0183] (20)

[0184] in, As for the degree of conflict, when At that time, the decision to execute the ruling is implemented, and only the decision to execute the ruling is retained. The largest modality is used as the primary evidence, while the others are used as secondary evidence.

[0185] Furthermore, for each fusion instance Output instance-level uncertainty indicators This indicator is calculated using formula (21) based on the average entropy of the fusion probability map within the instance region:

[0186] (twenty one)

[0187] in, The final output is the multi-instance segmentation result. } and the corresponding uncertainty { }

[0188] This embodiment achieves complementary advantages and weighted integration of multimodal evidence through dual weighting of credibility and modality sensitivity, thereby improving the accuracy and robustness of segmentation results. Simultaneously, its built-in conflict detection and adjudication strategy provides a fault-tolerant mechanism, automatically switching to a more conservative and reliable decision-making mode when multimodal results are inconsistent. This effectively prevents erroneous fusion in cases of contradictory evidence, ensuring the clinical reliability and decision security of the final output.

[0189] S207: Outputs a multi-instance segmentation result consisting of the fused segmentation results of all targets.

[0190] This embodiment acquires multimodal images and completes spatial registration, providing a consistent coordinate basis for subsequent cross-modal analysis. By standardizing and generating prompts for target segmentation concepts, it improves the model's understanding and accuracy in identifying complex medical terms. A credibility-gated model scores and filters candidate results from each modality, effectively eliminating low-quality and unreliable segmentation proposals. By calculating cross-modal instance matching costs and establishing correspondences, it reliably correlates segmentation results of the same real target across different modalities. Furthermore, it performs weighted fusion based on credibility and modality weights, and activates an adjudication strategy when conflicts exceed a threshold, effectively integrating multi-source information and resolving inconsistencies between modalities. This improves the accuracy and consistency of multimodal medical image segmentation.

[0191] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0192] Figure 4 A schematic block diagram of a multimodal medical image segmentation device is shown, corresponding one-to-one with the multimodal medical image segmentation methods described in the above embodiments. For example... Figure 4 As shown, the functional modules of this multimodal medical image segmentation device are described in detail below:

[0193] The image registration module 31 is used to acquire multimodal medical images of the same subject, preprocess and spatially register each modality image to obtain a multimodal image set registered to the same reference space;

[0194] The text parsing module 32 is used to perform medical terminology standardization and structured parsing on the input target segmentation concept, and generate segmentation prompt information based on the parsing results;

[0195] The information segmentation module 33 is used to input the images and segmentation prompts of each modality in the multimodal image set into the basic segmentation model respectively, and obtain multiple candidate segmentation results corresponding to each modality and the confidence information of each candidate segmentation result;

[0196] The credibility assessment module 34 is used to assess the credibility of each candidate segmentation result based on the credibility gating model, obtain the credibility score of each candidate segmentation result, and filter according to the credibility score to obtain a set of credible instances under each modality.

[0197] The instance matching module 35 is used to calculate the matching cost between different modalities of trusted instances in the trusted instance set, and to perform instance matching based on the matching cost to form multiple instance groups, each instance group corresponding to a real anatomical or pathological target;

[0198] The weighted fusion module 36 is used to perform weighted fusion based on the credibility score of each trusted instance in the group and the sensitivity weight of its modality to obtain the fusion segmentation result of the target; when the segmentation results of each instance in the group conflict more than a threshold, an adjudication strategy is executed.

[0199] The result output module 37 is used to output a multi-instance segmentation result consisting of the fused segmentation results of all targets.

[0200] Specific limitations regarding the multimodal medical image segmentation device can be found in the limitations of the multimodal medical image segmentation method described above, and will not be repeated here. Each module in the aforementioned multimodal medical image segmentation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0201] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 5 , Figure 5 This is a basic structural block diagram of the computer device in this embodiment.

[0202] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components connected to the memory 41, processor 42, and network interface 43 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0203] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0204] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D-interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as the program code of a multimodal medical image segmentation method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0205] In some embodiments, the processor 42 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run program code stored in the memory 41 or process data, for example, to run program code for a multimodal medical image segmentation method.

[0206] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0207] This application also provides another embodiment, namely, a computer-readable storage medium storing an interface display program that can be executed by at least one processor to cause the at least one processor to perform the steps of the multimodal medical image segmentation method described above.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0209] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A multimodal medical image segmentation method, characterized in that, include: Acquire multimodal medical images of the same subject, preprocess and spatially register each modality of the images to obtain a multimodal image set registered to the same reference space; The input target segmentation concept is standardized and structured using medical terminology, and segmentation prompts are generated based on the parsing results. The images of each modality in the multimodal image set and the segmentation prompt information are respectively input into the basic segmentation model to obtain multiple candidate segmentation results corresponding to each modality and the confidence information of each candidate segmentation result; Based on the credibility gating model, the credibility of each candidate segmentation result is evaluated to obtain a credibility score for each candidate segmentation result. Then, based on the credibility score, a set of credible instances under each modality is obtained. Calculate the matching cost between different modalities of trusted instances in the trusted instance set, and perform instance matching based on the matching cost to form multiple instance groups, each instance group corresponding to a real anatomical or pathological target; The target segmentation result is obtained by weighted fusion based on the credibility score of each trusted instance in the group and the sensitivity weight of its modality; when the segmentation results of each instance in the group conflict more than a threshold, an adjudication strategy is executed. The output consists of a multi-instance segmentation result composed of the fused segmentation results of all targets.

2. The method according to claim 1, characterized in that, The credibility-gated model is used to evaluate the credibility of each candidate segmentation result, and the credibility score of each candidate segmentation result includes: For each candidate segmentation result, the image semantic features and mask shape features of the candidate segmentation result are extracted, and a conditional vector containing information on concept existence, image quality and modality sensitivity is constructed. The image semantic features, mask shape features, and conditional vector are input into the credibility gating model to calculate the credibility score of the candidate segmentation result; The credibility score is compared with a preset threshold to filter out the set of credible instances.

3. The method according to claim 2, characterized in that, The extraction of image semantic features and mask shape features from the candidate segmentation results includes: The image encoder of the basic segmentation model is used to obtain the deep embedding feature map of the input image; The image semantic features are obtained by performing region pooling on the deep embedding feature map using a binary mask of the candidate segmentation results.

4. The method according to claim 2, characterized in that, The step of inputting the image semantic features, mask shape features, and conditional vector into the confidence gating model to calculate the confidence score of the candidate segmentation result includes: The image semantic features, mask shape features, and conditional vector are input into a conditional variational autoencoder; Obtain the reconstruction error and posterior uncertainty of the latent variables from the output of the conditional variational autoencoder; The confidence score is calculated using a parameterized fusion function based on the reconstruction error, the posterior uncertainty, and the conditional vector.

5. The method according to claim 2, characterized in that, The matching cost between different modal trusted instances in the set of trusted instances includes: Calculate the mask space similarity between any two trustworthy instances from different modalities; Calculate the feature space distance between the image semantic features of the two trusted instances; The matching cost is obtained by weighted summation of the mask space similarity and the feature space distance.

6. The method according to claim 1, characterized in that, The weighted fusion based on the credibility scores of each trusted instance within the group and the sensitivity weights of its respective modality yields the following fusion segmentation results for the target: Assign a fusion weight to each trusted instance within the instance group. The fusion weight is the product of the trusted instance's trust score and the sensitivity weight of its modality. Using the fusion weights, the segmentation probability maps of each trusted instance are weighted and averaged to obtain the fusion probability map of the target. The fusion probability map is binarized to obtain the fusion segmentation result.

7. The method according to claim 6, characterized in that, The adjudication strategy is as follows: when a conflict occurs, during the weighted fusion process, only the segmentation probability map corresponding to the credible instance with the largest fusion weight is adopted as the primary evidence for generating the fusion probability map.

8. A multimodal medical image segmentation device, characterized in that, include: The image registration module is used to acquire multimodal medical images of the same subject, preprocess and spatially register each modality of image to obtain a multimodal image set registered to the same reference space; The text parsing module is used to perform medical terminology standardization and structured parsing on the input target segmentation concept, and generate segmentation prompt information based on the parsing results; The information segmentation module is used to input the images of each modality in the multimodal image set and the segmentation prompt information into the basic segmentation model respectively, and obtain multiple candidate segmentation results corresponding to each modality and the confidence information of each candidate segmentation result; The credibility assessment module is used to assess the credibility of each candidate segmentation result based on the credibility gating model, obtain a credibility score for each candidate segmentation result, and filter according to the credibility score to obtain a set of credible instances under each modality. The instance matching module is used to calculate the matching cost between trusted instances of different modalities in the trusted instance set, and to perform instance matching based on the matching cost to form multiple instance groups, each instance group corresponding to a real anatomical or pathological target; The weighted fusion module is used to perform weighted fusion based on the credibility score of each trusted instance in the group and the sensitivity weight of its modality to obtain the fused segmentation result of the target; when the segmentation results of the instances in the group conflict more than a threshold, an adjudication strategy is executed. The results output module is used to output a multi-instance segmentation result consisting of the fused segmentation results of all targets.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal medical image segmentation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal medical image segmentation method as described in any one of claims 1 to 7.

Images