Medical image segmentation method and system based on evidence-driven visual language model

Abstract

The present invention discloses a medical image segmentation method and system based on an evidence-driven visual language model, and relates to the technical field of medical image segmentation. The method comprises the following steps: obtaining the original medical image to be segmented; constructing a multimodal segmentation model, calculating the similarity loss using a deviation-variance decomposition method based on a differential matrix, introducing uncertainty perspectives to represent the visual evidence embedding and the textual evidence embedding as visual perspectives and textual perspectives, respectively, and calculating the perspective loss, and calculating the segmentation loss using the segmentation difference between the visual-text fusion evidence and the true mask; optimizing the parameters of the multimodal segmentation model using the overall loss, and performing image segmentation on the original medical image to be segmented using the multimodal segmentation model. The present invention introduces evidence learning into the visual language model, and estimates the modal gap between the image and the text by aggregating the image-text perspectives converted from the evidence, thereby achieving multimodal fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a medical image segmentation method and system based on an evidence-driven visual language model. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In the medical field, medical images contain a large amount of image and text information. Existing technologies fail to deeply mine and map these two elements, resulting in inaccurate image segmentation and the inability to accurately identify lesion areas, hindering doctors' diagnosis. Therefore, accurate medical image segmentation is crucial for disease diagnosis.

[0004] Visual-language models (VLMs) can learn visual-language representations by adopting a BERT-like architecture and a contrastive learning paradigm. Specifically, VLMs utilize contrastive learning to optimize the model by distinguishing the similarities between different images and text, enabling the model to better capture the complex relationships between images and text. However, current VLMs suffer from a modality gap, resulting in poor performance on downstream tasks. The modality gap specifically refers to the difference between images and text. In medical image segmentation tasks based on VLMs, image and text representations often cluster into two distinct groups. Due to the inherent differences between image and text data, there is a significant modality gap between them. This modality gap makes it difficult to accurately measure the similarity between image and text, which in turn affects modality fusion and leads to poor segmentation performance. Existing methods primarily bridge the modality gap by extracting cross-modal features using two independent encoders. These features are then embedded into a latent common space using carefully designed objective functions to achieve modality invariance. However, these methods have shortcomings: they rely on shallow interactions between modalities and fail to fully address the gap between them. The narrowness of the embedding space limits the ability to express the complex semantic relationships between images and texts, so this shallow modal interaction method cannot effectively capture and fuse the deep associations between images and texts. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a medical image segmentation method and system based on an evidence-driven visual language model, introduce evidence learning into the visual language model, and estimate the modal gap between image and text by aggregating the image-text viewpoints after evidence conversion, thereby realizing multimodal fusion to improve the accuracy and robustness of medical image segmentation.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A first aspect of the present invention provides a medical image segmentation method based on an evidence-driven visual language model, comprising the following steps:

[0008] Obtaining an original medical image to be segmented, where the original medical image contains image information and text information;

[0009] A multimodal segmentation model is constructed. Based on the evidence learning algorithm, a visual language model is used to calculate the overall loss in the image segmentation process. The overall loss includes similarity loss, viewpoint loss, and segmentation loss. The image information used for training is encoded into visual evidence embeddings and the text information is encoded into text evidence embeddings. The similarity loss of the visual evidence embeddings and the text evidence embeddings is calculated using a bias-variance decomposition method based on a difference matrix. The visual evidence embeddings and the text evidence embeddings are represented as visual viewpoints and text viewpoints, respectively, by introducing uncertain viewpoints. The viewpoint loss is calculated based on the visual viewpoints and text viewpoints. The visual evidence embeddings and the text evidence embeddings are fused to obtain visual-text fusion evidence. The segmentation loss is calculated using the segmentation difference between the visual-text fusion evidence and the true mask.

[0010] The overall loss is used to optimize the parameters of the multimodal segmentation model, and the finally trained multimodal segmentation model is used to perform image segmentation on the original medical images to be segmented.

[0011] Furthermore, the specific steps of encoding the image information used for training into visual evidence embedding and encoding the text information into text evidence embedding are as follows:

[0012] Use visual encoder to encode image information and obtain visual evidence embedding;

[0013] Use a text encoder to encode text information and obtain text tag embedding;

[0014] The visual evidence embedding is combined with the textual token embedding through a cross-attention module to obtain the textual evidence embedding.

[0015] Furthermore, after obtaining the visual evidence embedding and the textual evidence embedding, the visual evidence embedding and the textual evidence embedding are refined. The specific steps are as follows:

[0016] The non-local self-attention block is used to learn the cross-modal evidence affinity of visual evidence embedding and textual evidence embedding, and the visual evidence affinity graph and textual evidence affinity graph are obtained;

[0017] The self-attention module is used to synthesize the visual evidence affinity graph and the textual evidence affinity graph to obtain a global cross-modal affinity graph;

[0018] Affine the global cross-modal affinity graph on the visual evidence affinity graph and the textual evidence affinity graph to obtain the refined video cross-modal evidence embedding and the refined text cross-modal evidence embedding respectively.

[0019] Furthermore, the specific steps of calculating the similarity loss of visual evidence embedding and text evidence embedding using the bias-variance decomposition method based on the difference matrix are as follows:

[0020] Introducing the uncertainty of viewpoints, the bias-variance decomposition method is performed using the difference matrix to calculate the inconsistency between the visual evidence embedding and the textual evidence embedding, and the inconsistency differential loss is obtained;

[0021] Compute the InfoNCE loss of the similarity matrix of the visual evidence embedding and the similarity matrix of the textual evidence embedding;

[0022] The similarity loss is obtained by adding the inconsistency difference loss and the InfoNCE loss.

[0023] Furthermore, we introduce the uncertainty of viewpoints and use the difference matrix to perform the bias-variance decomposition method to calculate the inconsistency between the visual evidence embedding and the textual evidence embedding. The specific steps are:

[0024] Calculate the similarity between visual evidence embedding and text evidence embedding based on the uncertainty of viewpoints, and obtain the similarity matrix of visual evidence embedding and the similarity matrix of text evidence embedding;

[0025] The bias-variance decomposition method is performed using the difference matrix to learn the evidence difference similarity of the similarity matrix of visual evidence embedding and the similarity matrix of textual evidence embedding, and the inconsistency difference loss is obtained.

[0026] Furthermore, the specific steps for calculating the segmentation loss using the segmentation difference between the visual text fusion evidence and the true mask are as follows:

[0027] Using a decoder to decode the refined video cross-modal evidence embedding and the refined text cross-modal evidence embedding to obtain decoded visual evidence and decoded text evidence;

[0028] Fusing the decoded visual evidence and the decoded text evidence to obtain visual-text fusion evidence;

[0029] A segmentation loss is introduced between the visual-text fusion evidence and the ground-truth mask.

[0030] Furthermore, the specific steps for calculating the opinion loss based on visual opinion and textual opinion are:

[0031] The decoded visual evidence and the decoded text evidence are represented as visual opinions and text opinions respectively using the Dirichlet distribution mapping method;

[0032] Fusing visual and textual views to obtain an aggregated view of visual and textual views;

[0033] The visual opinion loss is calculated based on the visual opinion, the textual opinion loss is calculated based on the textual opinion, and the aggregate loss is calculated based on the aggregated opinion of the visual and textual opinions. The visual opinion loss, textual opinion loss and aggregate loss are added together to get the opinion loss.

[0034] A second aspect of the present invention provides a medical image segmentation system based on an evidence-driven visual language model, comprising:

[0035] A data acquisition module is configured to acquire an original medical image to be segmented, where the original medical image includes image information and text information;

[0036] A model training module is configured to build a multimodal segmentation model, calculate the overall loss in the image segmentation process using a visual language model based on an evidence learning algorithm, wherein the overall loss includes similarity loss, viewpoint loss, and segmentation loss, encode image information used for training into visual evidence embedding and encode text information into text evidence embedding, calculate the similarity loss of the visual evidence embedding and the text evidence embedding using a bias-variance decomposition method based on a difference matrix, express the visual evidence embedding and the text evidence embedding as visual viewpoint and text viewpoint, respectively, by introducing uncertainty viewpoints, calculate the viewpoint loss based on the visual viewpoint and the text viewpoint, fuse the visual evidence embedding and the text evidence embedding to obtain visual-text fusion evidence, and calculate the segmentation loss using the segmentation difference between the visual-text fusion evidence and the true mask;

[0037] The image segmentation module is configured to optimize the parameters of the multimodal segmentation model using the overall loss, and perform image segmentation on the original medical image to be segmented using the finally trained multimodal segmentation model.

[0038] A third aspect of the present invention provides a medium having a program stored thereon, which, when executed by a processor, implements the steps of the medical image segmentation method based on the evidence-driven visual language model as described in the first aspect of the present invention.

[0039] The fourth aspect of the present invention provides a device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the medical image segmentation method based on the evidence-driven visual language model as described in the first aspect of the present invention are implemented.

[0040] One or more of the above technical solutions have the following beneficial effects:

[0041] This paper discloses a medical image segmentation method and system based on an evidence-driven visual language model. An evidence affinity graph generator (EAMG) is proposed to refine modality-specific evidence embeddings by learning a global cross-modal affinity graph, thereby collecting complementary cross-modal evidence. Furthermore, evidence difference similarity learning (EDSL) is proposed to collect consistent cross-modal evidence by analyzing the inconsistency of bidirectional similarity matrix changes based on bias-variance decomposition. Finally, subjective logic is used to map the collected evidence into opinions, and a combination rule based on the Dempster-Shafer theory is introduced for opinion aggregation and modality gap measurement to achieve full modality fusion.

[0042] This paper proposes a new visual language model paradigm, incorporating evidence-based learning into visual language models for the first time. By aggregating cross-modal insights to estimate the modal gap, this approach bridges the modality gap in cross-modal fusion between images and text. Through simulations, in some embodiments, this paper can achieve accurate segmentation of pneumonia lesions, lung infection areas, and breast cancer lesions.

[0043] In this paper, we propose an evidence affinity graph generator to collect complementary cross-modal evidence and refine the two modality-specific evidence embeddings by learning a global cross-modal affinity graph.

[0044] In this paper, an evidence difference similarity learning is proposed to improve the consistency between cross-modal evidence embeddings by measuring the variation inconsistency of the similarity matrix.

[0045] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 is a flowchart of a medical image segmentation method based on an evidence-driven visual language model in Example 1 of the present invention;

[0048] Figure 2 This is an overall framework diagram of the medical image segmentation method based on the evidence-driven visual language model in Example 1 of the present invention;

[0049] Figure 3 This is a framework diagram of an evidence affinity graph generator for cross-modal evidence affinity learning and integration in Example 1 of the present invention;

[0050] Figure 4This is a framework diagram of evidence difference similarity learning based on bias-variance decomposition of the difference matrix in Example 1 of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations;

[0053] Example 1:

[0054] Embodiment 1 of the present invention provides a medical image segmentation method based on an evidence-driven visual language model, which bridges the cross-modal gap between images and text by introducing evidence learning. Specifically, an evidence affinity graph generator (EAMG) is proposed to refine modality-specific evidence embeddings by aggregating cross-modal evidence and learning a global cross-modal affinity graph, thereby collecting complementary cross-modal evidence. In addition, evidence difference similarity learning (EDSL) is proposed to collect consistent cross-modal evidence through the inconsistency of changes in the bidirectional similarity matrix based on bias-variance decomposition, and the inconsistency of changes in the similarity matrix is ​​used to improve the consistency of cross-modal evidence embeddings, thereby further improving the fusion effect between modalities. Finally, subjective logic is used to map the collected evidence into opinions, and a combination rule based on the Dempster-Shafer theory is introduced to aggregate opinions and measure the modal gap to achieve full modal fusion.

[0055] The overall inventive concept of this embodiment is to improve segmentation accuracy by combining image and text information. Its core concept is to efficiently fuse visual and text evidence through multiple modules to perform cross-modal reasoning and optimization. Figure 1 and Figure 2As shown, image-text evidence embeddings are first extracted. Visual evidence embeddings are extracted from the original image using a visual encoder based on the U-Net architecture, while textual evidence embeddings are extracted from the text description using a text encoder based on the BioClinicalBERT architecture. To effectively fuse image and text, a cross-attention module is used to combine the visual and textual embeddings. Subsequently, the generated evidence affinity graph is used to enhance the complementarity between the image and text evidence. The evidence affinity graph generator module learns the affinity of visual and textual evidence using a self-attention mechanism, generating a global cross-modal pixel-level affinity graph to further refine the evidence embeddings. A differential evidence similarity learning loss is then designed to enhance the consistency between image and textual evidence. The evidence differential similarity learning module optimizes the alignment of image and textual evidence using a differential matrix, improving cross-modal alignment accuracy. The image-textual evidence is then decoded. The refined visual and textual evidence is fused through the decoding module. A segmentation loss is introduced between the fused visual-textual evidence and the ground-truth mask to optimize visual-textual joint reasoning. Finally, the image-textual evidence is converted into image-textual viewpoints. Image-textual viewpoints are aggregated according to the Dempster-Shafer theory, and the aggregated viewpoint loss is calculated. The model aggregates visual and textual insights, quantifying the gap between modalities and further enhancing reasoning capabilities. Finally, it jointly optimizes the differential evidence similarity learning loss, the aggregated insight loss, and the segmentation loss to train a multimodal segmentation model. This entire framework improves image segmentation accuracy by optimizing the loss functions of multiple modules. It has been applied to medical image segmentation tasks, achieving efficient collaborative processing of image and text information.

[0056] The specific steps include:

[0057] Step 1: Obtain the original medical image to be segmented. The original medical image contains image information and text information.

[0058] Step 2: Build a multimodal segmentation model and use the visual language model based on the evidence learning algorithm to calculate the overall loss in the image segmentation process.

[0059] Step 2.1: Encode the image information used for training into visual evidence embeddings and the text information into text evidence embeddings.

[0060] Step 2.1.1: Use the visual encoder to encode the image information and obtain the visual evidence embedding.

[0061] In a specific embodiment, for an image, the original image is fed into a visual encoder based on a U-Net architecture to directly extract visual evidence embeddings. In the visual encoder, the image is processed through multiple convolutional layers and pooling layers to gradually extract deeper visual features. These features are encoded as visual evidence embeddings, which can effectively represent the key information in the image.

[0062] Step 2.1.2: Use the text encoder to encode the text information and obtain the text token embedding.

[0063] In a specific embodiment, for text, the corresponding text description is fed into a text encoder based on the BioClinicalBERT architecture, and the visual evidence embedding uses a cross-attention mechanism to process the text tag embeddings encoded by the text encoder. These text embeddings contain grammatical and semantic information of the text description, which helps to understand the context of the medical image.

[0064] Step 2.1.3: Combine the visual evidence embedding with the textual token embedding through a cross-attention module to obtain the textual evidence embedding.

[0065] In one specific implementation, to achieve effective fusion between images and text, visual evidence embeddings are combined with text tag embeddings via a cross-attention module. This cross-attention module allows the model to dynamically reference and utilize related text information while processing the image, thereby generating more accurate text evidence embeddings.

[0066] Specifically, the image and text are fed into the visual encoder and text encoder respectively to extract the image evidence embedding and text evidence embedding. The specific process of step 2.1 is as follows:

[0067] Given an image-text pair {V,T}, the encoding path of U-Net and BioClinicalBERT is used as the visual encoder and text encoders To extract visual evidence embeddings and text evidence embedding in for It is directly Extraction, that is: for It is obtained through the Cross-Attention Module (CA). Formally, for visual evidence embedding let Processing by Encoded text token embedding x T , and then calculate its corresponding cross-modal text evidence embedding Right now:

[0068]

[0069] in, They are all learnable matrices, d is the characteristic dimension of Q, K, and V matrices, is a set of real numbers, α V2T Refers to the attention matrix from image to text, and V2T (vision to text) means image to text.

[0070] Step 2.2: After obtaining the visual evidence embedding and textual evidence embedding, the visual evidence embedding and textual evidence embedding are refined. The specific steps are as follows:

[0071] Step 2.2.1: Use the non-local self-attention block to learn the cross-modal evidence affinity of visual evidence embedding and textual evidence embedding, and obtain the visual evidence affinity graph and textual evidence affinity graph.

[0072] Step 2.2.2: Use the self-attention (SA) module to synthesize the visual evidence affinity graph and the textual evidence affinity graph to obtain a global cross-modal affinity graph.

[0073] In a specific implementation, in order to learn multimodal evidence affinity, a self-attention (SA) module is used to integrate the two modality-specific affinity maps with self-attention to synthesize two modality-specific affinity maps. By integrating the two learned visual-perception evidence affinities and text-perception evidence affinities, a global cross-modal pixel-level affinity map is generated for evidence embedding refinement, thereby strengthening the complementarity between images and text.

[0074] Step 2.2.3: Affine the global cross-modal affinity graph on the visual evidence affinity graph and the textual evidence affinity graph respectively to obtain the refined video cross-modal evidence embedding and the refined text cross-modal evidence embedding.

[0075] Specifically, the evidence affinity graph generation module integrates two learned visual-aware and text-aware evidence affinities to generate a global cross-modal pixel-level affinity graph for evidence embedding refinement, thereby strengthening the complementarity between image and text. Figure 3 As shown, the specific process of step 2.2 is as follows:

[0076] The non-local self-attention block (NonLocal) is used to capture the semantic correlation of spatial positions and learn the correlation between modalities based on the similarity between the feature vectors of any two positions.

[0077]

[0078] In particular, for The input image feature is a tensor of size H×W×D, which is encoded into a triplet Q, K, V through three 1×1 convolutional layers, and then such a triplet is flattened to size HW×D. The visual evidence affinity graph is calculated by the dot product of Q and K Each row Represents the similarity between a spatial location and all other spatial locations. Text Evidence Affinity Graph It is generated in a similar way to the visual evidence affinity graph. In order to learn multimodal evidence affinity, a self-attention (SA) module is used to synthesize two modality-specific affinity graphs, and then the SA module generates two spatial attention maps. Visual Evidence Affinity Diagram and text evidence affinity diagram Aggregate into a global cross-modal affinity graph

[0079]

[0080] in, are the two learned spatial attention maps. In order to refine the two modality-specific evidence embeddings respectively and Upper affine global cross-modal affinity graph A evi , thus obtaining two refined cross-modal evidence embeddings and

[0081]

[0082]

[0083] where affine(·) refers to the evidence affine operation and the operation used with evidence embedding is the Hadamard product.

[0084] Visual evidence embedding via evidence affine operator and affinity diagram A evi Computed refined affine visual evidence embedding It can be expressed as:

[0085]

[0086] Similarly, based on text evidence embedding and affinity diagram A evi Computed refined affine visual evidence embedding It can be expressed as:

[0087]

[0088] Where h and w refer to the width and height of the affinity graph. The evidence affinity graph generator refines cross-modal evidence embedding by learning a global pixel-level affinity graph, enhancing the complementarity of cross-modal evidence learning.

[0089] Step 2.3: Calculate the similarity loss between the visual evidence embedding and the textual evidence embedding using the bias-variance decomposition method based on the difference matrix.

[0090] Step 2.3.1: Introduce the uncertainty of viewpoints and use the difference matrix to perform the bias-variance decomposition method to calculate the inconsistency between the visual evidence embedding and the textual evidence embedding, and obtain the inconsistency difference loss.

[0091] In one specific implementation, the visual evidence embeddings and the textual evidence embeddings are fed into an evidence difference similarity learning module. This module performs a bias-variance decomposition based on the difference matrix to learn the variation inconsistencies between the two cross-modal similarity matrices, thereby improving the alignment between the image and textual evidence embeddings. To further ensure the reliability of the similarity matrix, visual viewpoint uncertainty and textual viewpoint uncertainty are also applied to the visual evidence embeddings and textual evidence embeddings, respectively.

[0092] Step 2.3.1.1: Calculate the similarity between the visual evidence embedding and the textual evidence embedding based on the uncertainty of the viewpoints to obtain the similarity matrix of the visual evidence embedding and the similarity matrix of the textual evidence embedding.

[0093] Step 2.3.1.2: Use the difference matrix to perform the bias-variance decomposition method to learn the evidence difference similarity of the similarity matrix of the visual evidence embedding and the similarity matrix of the textual evidence embedding, and obtain the inconsistency difference loss.

[0094] Specifically, such as Figure 4 As shown, step 2.3.1 includes: the evidence difference similarity learning module performs bias variance decomposition based on the difference matrix to learn the variation inconsistency between the two bidirectional cross-modal similarity matrices, thereby enhancing the strong alignment between the image and text evidence embeddings. To further ensure the reliability of the similarity matrix, the visual view uncertainty u V and text view uncertainty u T Applied to visual evidence embedding and text evidence embedding So for and Their evidence similarity matrix s ij and It can be calculated by the following formula:

[0095]

[0096]

[0097] Among them, s ij represents the similarity matrix of the visual evidence embedding, represents the similarity matrix of text evidence embedding, cos(·) measures the cosine similarity, and i and j represent the i-th image and j-th text respectively. and They represent two bidirectional cross-modal similarity matrices, respectively, where B is the batch size.

[0098] In order to measure the similarity between two matrices [S V2T ,S T2V ] changes between the inconsistency, from S T2V Subtract S from V2T To obtain their difference matrix S diff , then based on S diff Perform bias-variance decomposition.

[0099] S diff =f BVD (S V2T ,S T2V ) (10).

[0100] The bias-variance decomposition is defined as follows: Let S V2T and S T2V are the two-way similarity matrices respectively. Deviation-variance decomposition function f BVD The derivation is as follows:

[0101]

[0102] in, Var(·) and Cov(·) represent the expectation, variance, and covariance operations, respectively. The calculated deviation and variance measure the inconsistency of the changes between the two cross-modal similarity matrices. Therefore, Inconsistency differential loss as similarity matrix

[0103] Step 2.3.2: Compute the InfoNCE loss of the similarity matrix of the visual evidence embedding and the similarity matrix of the textual evidence embedding.

[0104] The InfoNCE (Information Noise-Contrastive Estimation) loss of the above two cross-modal similarity matrices is calculated to maximize the preservation of the mutual information between the real image-text pairs.

[0105]

[0106] in, is the image-to-text InfoNCE loss, is the text-to-image InfoNCE loss, and τ is the temperature hyperparameter.

[0107] Step 2.3.3: Add the inconsistency difference loss and InfoNCE loss to get the similarity loss Used to enforce strong matching between image and text evidence embeddings.

[0108] Specifically, the overall objective of EDSL is the sum of InfoNCE loss and inconsistency loss.

[0109]

[0110] Among them, λ1 and λ2 are hyperparameters used to balance the two losses.

[0111] EDSL performs bias-variance decomposition based on the difference matrix and learns the consistency of the cross-modal similarity matrix, thereby enhancing the consistency of cross-modal evidence.

[0112] Step 2.4: Fuse the visual evidence embedding and the textual evidence embedding to obtain the visual-textual fusion evidence, and use the segmentation difference between the visual-textual fusion evidence and the true mask to calculate the segmentation loss.

[0113] In a specific implementation, the refined cross-modal visual evidence embedding and textual evidence embedding are fed into the visual decoder and textual decoder respectively to obtain decoded visual evidence and textual evidence, which are then fused together and segmentation loss is introduced between the fused visual-textual evidence and the true mask. The purpose is to help the model better perform joint reasoning of images and texts during training and optimize the visual-text fusion of the model, thereby improving the accuracy of image segmentation tasks, especially when combined with textual information.

[0114] Step 2.4.1: Use decoder to embed refined video cross-modal evidence and refined textual cross-modal evidence Embedded for decoding, to obtain decoded visual evidence and decoded textual evidence

[0115] Step 2.4.2: Fuse the decoded visual evidence and the decoded textual evidence by adding them together to obtain visual-textual fusion evidence.

[0116] Step 2.4.3: Introduce a segmentation loss between the visual-text fusion evidence and the pre-annotated ground-truth masks.

[0117] Specifically, the segmentation loss between the fused visual-textual evidence and the true template is introduced

[0118] Where m represents the mth pixel in the mask. W and H are the width and height of the image, is the cross entropy loss, is the evidence for the mth pixel in the image, is the evidence of the mth pixel in the text, y m is the value of the mth pixel in the true mask.

[0119] Step 2.5: Represent the visual evidence embedding and textual evidence embedding as visual opinion and textual opinion respectively by introducing uncertainty opinion, and calculate the opinion loss based on visual opinion and textual opinion.

[0120] In one specific implementation, to measure modality gap based on aggregated opinion, the decoded visual and textual evidence are represented as visual opinion and textual opinion to quantify Dirichlet distribution uncertainty. Each opinion reflects the reasoning result of the evidence from that modality under a given problem. Ultimately, the visual opinion and textual opinion are aggregated into an overall opinion, which is a comprehensive reasoning result that combines visual and textual information.

[0121] Step 2.5.1: Use the Dirichlet distribution mapping method to represent the decoded visual evidence and the decoded textual evidence as visual opinions and textual opinions respectively.

[0122] Step 2.5.2: Fuse the visual viewpoint and the textual viewpoint to obtain the aggregated viewpoint of the visual and textual views.

[0123] Step 2.5.3: Calculate the visual opinion loss based on the visual opinion, calculate the textual opinion loss based on the textual opinion, calculate the aggregate loss based on the aggregated opinion of the visual and textual opinions, and add the visual opinion loss, textual opinion loss, and aggregate loss to get the opinion loss.

[0124] Specifically, step 2.5 includes: in order to measure the modality gap based on the aggregated viewpoint, the decoded visual evidence and textual evidence Represented as visual point o V and text view T To quantify Dir(μ V |α V ) and Dir(μ T |α T ) in the distribution uncertainty.

[0125] In evidence-based learning, it is often necessary to infer the probability distribution of a multi-class problem based on data. The Dirichlet distribution can be used as a prior distribution for these classes. It helps smooth the data and enables the model to handle sparsity problems.

[0126] According to subjective logic, a principle method for probabilistic reasoning under uncertainty, the visual Dirichlet distribution Dir(μ V |α V ) by evidence Determine, where C is the number of categories. V =e V +1 to derive the distribution parameter. Then, the visual Dirichlet distribution is mapped to the visual point of view satisfy:

[0127]

[0128] in, is the confidence quality of category c, is the sum of Dirichlet distributions, Measuring uncertainty in a Dirichlet distribution.

[0129] Final predicted pixel-level probability is the expectation of the Dirichlet distribution, that is where μ V is the original predicted probability. Assume that pixel-level samples cannot provide any evidence for decision making, that is, e V = 0. According to a V , S V and μ V Definition of uncertainty μ V is negatively correlated with the sum of evidence. Therefore, this pixel-level sample will produce high uncertainty. The reasoning process of text opinion o={b1,b2,...,b C ,u}=o V ⊕o T The reasoning process is the same as that of vision.

[0130] To obtain the aggregated view based on visual and textual views, following the Dempster-Schafer evidence theory, we use the belief fusion operator to aggregate the visual views o V and text view T , specifically, for and Aggregate viewpoint o = {b1, b2, ..., b C ,u}=o V ⊕o T Derived from the following formula:

[0131]

[0132] in, is the normalization factor.

[0133] To calculate the loss of visual, textual and aggregated views, an integrated cross entropy loss is used. The view of visual loss is given by:

[0134]

[0135] Where y is the one-hot label and m is the digamma function. The overall view loss for visual, textual and combined views is given by:

[0136]

[0137] in, Indicates loss of perspective, and The realization of Same. In cross-modal joint reasoning, using Measuring modal gaps. Modal gaps refer to the difference between features of two modalities, such as the difference between image and text features. Measuring and reducing modal gaps through evidence learning helps enhance consistency between modalities and improve multimodal fusion performance.

[0138] Step 2.6: The overall loss includes similarity loss, viewpoint loss and segmentation loss. Therefore, the overall loss is expressed as:

[0139]

[0140] in, is the overall loss, and ω1, ω2, and ω3 are hyperparameters that balance the three objective functions.

[0141] Step 3: Optimize the parameters of the multimodal segmentation model using the overall loss, and use the trained multimodal segmentation model to segment the original medical image. The model trained in this example is applied to medical images of pneumonia, lung infection, and breast cancer, achieving accurate segmentation of pneumonia lesions, lung infection areas, and breast cancer lesions.

[0142] In a specific embodiment, the parameters of the learnable visual encoder, visual decoder, and text decoder are optimized based on the visual opinion loss, text opinion loss, aggregate opinion loss, segmentation loss, and the loss of the evidence difference similarity learning module. Specifically, the optimization refers to backpropagation using the overall loss to optimize the encoder and decoder parameters. It should be noted that the text encoder uses a pre-trained encoder. During training, the parameters of the text encoder are frozen, and therefore no parameter optimization is performed on the text encoder.

[0143] The final trained multimodal segmentation model is used for medical image segmentation tasks. The evidence-driven visual language model network can be trained in an end-to-end manner.

[0144] This method proposes a novel vision-language model paradigm—an evidence-driven vision-language model. Traditional cross-modal learning methods often face differences between modalities, which makes effective information fusion more difficult. This method innovatively introduces evidence-based learning techniques to address the current modality gap between images and text, thereby achieving more efficient cross-modal fusion. This not only strengthens the connection between images and text during the learning process, but also makes the integration of cross-modal data more intimate and efficient.

[0145] This method proposes an evidence affinity graph generation method to collect and integrate cross-modal evidence. This method further refines the evidence embeddings specific to each modality by learning a global cross-modal affinity graph. Specifically, this method uses a detailed mapping relationship to enable features from different modalities to complement each other, thereby improving the quality of cross-modal fusion. Furthermore, by strengthening the relevance of evidence, it effectively reduces information inconsistencies between different modalities and improves the accuracy of the fusion results.

[0146] The method proposed in this paper proposes an evidence difference similarity learning method to ensure that evidence across different modalities can maintain consistency. This method enhances the alignment between cross-modal evidence embeddings by measuring the inconsistency of changes in the similarity matrix. In other words, this method ensures the semantic consistency of image and text information by comparing the similarity of evidence across different modalities. Ultimately, the collected cross-modal evidence is converted into an estimate of the modal gap, providing an accurate basis for further modal integration.

[0147] Example 2:

[0148] A second embodiment of the present invention provides a medical image segmentation system based on an evidence-driven visual language model, including:

[0149] A data acquisition module is configured to acquire an original medical image to be segmented, where the original medical image includes image information and text information;

[0150] A model training module is configured to build a multimodal segmentation model, calculate the overall loss in the image segmentation process using a visual language model based on an evidence learning algorithm, wherein the overall loss includes similarity loss, viewpoint loss, and segmentation loss, encode image information used for training into visual evidence embedding and encode text information into text evidence embedding, calculate the similarity loss of the visual evidence embedding and the text evidence embedding using a bias-variance decomposition method based on a difference matrix, express the visual evidence embedding and the text evidence embedding as visual viewpoint and text viewpoint, respectively, by introducing uncertainty viewpoints, calculate the viewpoint loss based on the visual viewpoint and the text viewpoint, fuse the visual evidence embedding and the text evidence embedding to obtain visual-text fusion evidence, and calculate the segmentation loss using the segmentation difference between the visual-text fusion evidence and the true mask;

[0151] The image segmentation module is configured to optimize the parameters of the multimodal segmentation model using the overall loss, and perform image segmentation on the original medical image to be segmented using the finally trained multimodal segmentation model.

[0152] Example 3:

[0153] A third embodiment of the present invention provides a medium having a program stored thereon. When the program is executed by a processor, the steps of the medical image segmentation method based on the evidence-driven visual language model as described in the first embodiment of the present invention are implemented.

[0154] Example 4:

[0155] Embodiment 4 of the present invention provides a device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, the steps in the medical image segmentation method based on the evidence-driven visual language model as described in Embodiment 1 of the present invention are implemented.

[0156] The steps involved in the above embodiments 2, 3 and 4 correspond to those in the method embodiment 1. For the specific implementation methods, please refer to the relevant description part of the embodiment 1.

[0157] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0158] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A medical image segmentation method based on evidence-driven visual language model, characterized in that: The following steps are involved: Obtaining an original medical image to be segmented, where the original medical image contains image information and text information; A multimodal segmentation model is constructed. Based on the evidence learning algorithm, a visual language model is used to calculate the overall loss in the image segmentation process. The overall loss includes similarity loss, viewpoint loss, and segmentation loss. The image information used for training is encoded into visual evidence embeddings and the text information is encoded into text evidence embeddings. The similarity loss of the visual evidence embeddings and the text evidence embeddings is calculated using a bias-variance decomposition method based on a difference matrix. The visual evidence embeddings and the text evidence embeddings are represented as visual viewpoints and text viewpoints, respectively, by introducing uncertain viewpoints. The viewpoint loss is calculated based on the visual viewpoints and text viewpoints. The visual evidence embeddings and the text evidence embeddings are fused to obtain visual-text fusion evidence. The segmentation loss is calculated using the segmentation difference between the visual-text fusion evidence and the true mask. The overall loss is used to optimize the parameters of the multimodal segmentation model, and the finally trained multimodal segmentation model is used to perform image segmentation on the original medical image to be segmented; The specific steps for calculating the similarity loss between visual evidence embedding and text evidence embedding using the bias-variance decomposition method based on the difference matrix are as follows: Introducing the uncertainty of viewpoints, the bias-variance decomposition method is performed using the difference matrix to calculate the inconsistency between the visual evidence embedding and the textual evidence embedding, and the inconsistency differential loss is obtained; Compute the InfoNCE loss of the similarity matrix of the visual evidence embedding and the similarity matrix of the textual evidence embedding; The similarity loss is obtained by adding the inconsistency difference loss and the InfoNCE loss; The specific steps of calculating the segmentation loss using the segmentation difference between the visual text fusion evidence and the true mask are: Using a decoder to decode the refined video cross-modal evidence embedding and the refined text cross-modal evidence embedding to obtain decoded visual evidence and decoded text evidence; The decoded visual evidence and the decoded text evidence are fused to obtain the visual-text fusion evidence; a segmentation loss is introduced between the visual-text fusion evidence and the true mask; The specific steps for calculating the opinion loss based on visual opinion and textual opinion are: The decoded visual evidence and the decoded text evidence are represented as visual opinions and text opinions respectively using the Dirichlet distribution mapping method; Fusing visual and textual views to obtain an aggregated view of visual and textual views; The visual opinion loss is calculated based on the visual opinion, the textual opinion loss is calculated based on the textual opinion, and the aggregate loss is calculated based on the aggregated opinion of the visual and textual opinions. The visual opinion loss, textual opinion loss and aggregate loss are added together to get the opinion loss.

2. The medical image segmentation method based on the evidence-driven visual language model according to claim 1, characterized in that: The specific steps for encoding image information for training into visual evidence embedding and text information into text evidence embedding are: Use visual encoder to encode image information and obtain visual evidence embedding; Use a text encoder to encode text information and obtain text tag embedding; The visual evidence embedding is combined with the textual token embedding through a cross-attention module to obtain the textual evidence embedding.

3. The medical image segmentation method based on the evidence-driven visual language model according to claim 2, characterized in that: After obtaining the visual evidence embedding and textual evidence embedding, the visual evidence embedding and textual evidence embedding are refined. The specific steps are as follows: The non-local self-attention block is used to learn the cross-modal evidence affinity of visual evidence embedding and textual evidence embedding, and the visual evidence affinity graph and textual evidence affinity graph are obtained; The self-attention module is used to synthesize the visual evidence affinity graph and the textual evidence affinity graph to obtain a global cross-modal affinity graph; Affine the global cross-modal affinity graph on the visual evidence affinity graph and the textual evidence affinity graph to obtain the refined video cross-modal evidence embedding and the refined text cross-modal evidence embedding respectively.

4. The medical image segmentation method based on the evidence-driven visual language model according to claim 1, characterized in that: The specific steps of introducing the uncertainty of viewpoints and using the difference matrix to perform the bias-variance decomposition method to calculate the inconsistency between the visual evidence embedding and the textual evidence embedding are as follows: Calculate the similarity between visual evidence embedding and text evidence embedding based on the uncertainty of viewpoints, and obtain the similarity matrix of visual evidence embedding and the similarity matrix of text evidence embedding; The bias-variance decomposition method is performed using the difference matrix to learn the evidence difference similarity of the similarity matrix of visual evidence embedding and the similarity matrix of textual evidence embedding, and the inconsistency difference loss is obtained.

5. A medical image segmentation system based on evidence-driven visual language model, characterized by: include: A data acquisition module is configured to acquire an original medical image to be segmented, where the original medical image includes image information and text information; A model training module is configured to build a multimodal segmentation model, calculate the overall loss in the image segmentation process using a visual language model based on an evidence learning algorithm, wherein the overall loss includes similarity loss, viewpoint loss, and segmentation loss, encode image information used for training into visual evidence embedding and encode text information into text evidence embedding, calculate the similarity loss of the visual evidence embedding and the text evidence embedding using a bias-variance decomposition method based on a difference matrix, express the visual evidence embedding and the text evidence embedding as visual viewpoint and text viewpoint, respectively, by introducing uncertainty viewpoints, calculate the viewpoint loss based on the visual viewpoint and the text viewpoint, fuse the visual evidence embedding and the text evidence embedding to obtain visual-text fusion evidence, and calculate the segmentation loss using the segmentation difference between the visual-text fusion evidence and the true mask; An image segmentation module is configured to optimize parameters of a multimodal segmentation model using an overall loss, and perform image segmentation on a raw medical image to be segmented using the finally trained multimodal segmentation model; The specific steps for calculating the similarity loss between visual evidence embedding and text evidence embedding using the bias-variance decomposition method based on the difference matrix are as follows: Introducing the uncertainty of viewpoints, the bias-variance decomposition method is performed using the difference matrix to calculate the inconsistency between the visual evidence embedding and the textual evidence embedding, and the inconsistency differential loss is obtained; Compute the InfoNCE loss of the similarity matrix of the visual evidence embedding and the similarity matrix of the textual evidence embedding; The similarity loss is obtained by adding the inconsistency difference loss and the InfoNCE loss; The specific steps of calculating the segmentation loss using the segmentation difference between the visual text fusion evidence and the true mask are: Using a decoder to decode the refined video cross-modal evidence embedding and the refined text cross-modal evidence embedding to obtain decoded visual evidence and decoded text evidence; The decoded visual evidence and the decoded text evidence are fused to obtain the visual-text fusion evidence; a segmentation loss is introduced between the visual-text fusion evidence and the true mask; The specific steps for calculating the opinion loss based on visual opinion and textual opinion are: The decoded visual evidence and the decoded text evidence are represented as visual opinions and text opinions respectively using the Dirichlet distribution mapping method; Fusing visual and textual views to obtain an aggregated view of visual and textual views; The visual opinion loss is calculated based on the visual opinion, the textual opinion loss is calculated based on the textual opinion, and the aggregate loss is calculated based on the aggregated opinion of the visual and textual opinions. The visual opinion loss, textual opinion loss and aggregate loss are added together to get the opinion loss.

6. A computer-readable storage medium, characterized in that Multiple instructions are stored therein, which are suitable for being loaded by a processor of a terminal device and executing the medical image segmentation method based on an evidence-driven visual language model according to any one of claims 1 to 4.

7. A terminal device, characterized in that: The invention comprises a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the medical image segmentation method based on the evidence-driven visual language model according to any one of claims 1 to 4.

Images