Cervical cancer MRI image automatic segmentation method based on multi-modal fusion

Abstract

The application provides a kind of cervical cancer MRI image automatic segmentation method based on multi-modal fusion, comprising the following steps: S1, obtains the tumor segmentation mask image of three modes of cervical cancer MRI image T1c, T2 and DWI by automatic prompt segmentation method, realizes the dynamic alignment of medical image features and segmentation task by the cross-modal attention interaction of task token and output token, and generates the segmentation mask image of each mode in combination with sparse, dense prompt;S2, the segmentation mask image of three modes is registered and feature is extracted, obtains the structured medical description of each mode segmentation mask image, and the structured medical description is fused by using large language model, and the tumor segmentation mask image after multi-modal fusion is obtained.The segmentation mask image of MRI multi-modal image is fused by introducing large language model in the application, and the tumor segmentation mask image after fusion is finally obtained by combining the segmentation characteristics of different modes, which greatly improves the segmentation precision of MRI image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an automatic segmentation method for cervical cancer MRI images based on multimodal fusion. Background Technology

[0002] Cervical cancer is one of the most common malignant tumors of the female reproductive system, with an increasing incidence rate and a trend towards younger patients. Magnetic resonance imaging (MRI) offers high image resolution for soft tissues, emits no ionizing radiation, and can image from different angles and sequences, clearly displaying the cervix, uterus, vagina, and surrounding tissue structures. It is commonly used in the clinical examination of cervical cancer.

[0003] In actual clinical treatment, radiologists need to manually and accurately delineate the area of ​​cervical cancer on the MRI imaging system, and then formulate a detailed radiotherapy and chemotherapy plan based on the size and location of the delineated cervical cancer. Due to the large number of MRI image sequences, manual segmentation is extremely time-consuming. Moreover, different physicians have subjective disagreements on the criteria for determining the "tumor boundary" (such as whether it includes microinvasive or edematous areas), and conflicting information provided by different modalities (such as DWI-positive but T2-negative areas) exacerbates the inconsistency in labeling.

[0004] Current technologies generally employ convolutional neural networks for medical image segmentation, delineating regions of interest layer by layer using the T1c, T2, and DWI modalities of cervical cancer MRI images. However, due to the different acquisition methods and characteristics of each modality, their contributions to segmentation vary, leading to inconsistent segmentation results across the three modalities, and poor segmentation accuracy for single-modality images. Specifically, T1c enhancement areas typically represent newly formed tumor tissue with rich blood supply or inflammatory areas, potentially missing non-enhancing necrotic areas or micro-infiltrations. T2 clearly displays the cervical anatomy, with tumors usually showing intermediate to high signal intensity, revealing the tumor's extent (including surrounding edema), but the boundaries may be blurred. DWI high b-value and low ADC values ​​represent densely packed cells with limited diffusion, serving as a sensitive marker of tumor activity, but may be affected by T2 penetration effects, necrosis, and hemorrhage, resulting in false positives / negatives. Summary of the Invention

[0005] This invention proposes an automatic segmentation method for cervical cancer MRI images based on multimodal fusion, which solves the problem of poor segmentation accuracy of single-modal images when using convolutional neural networks to segment MRI images in the prior art.

[0006] The technical solution of this invention is implemented as follows:

[0007] This invention provides an automatic segmentation method for cervical cancer MRI images based on multimodal fusion, comprising the following steps:

[0008] S1. Obtain tumor segmentation mask images of cervical cancer MRI images in three modalities: T1c, T2, and DWI through an automatic prompting segmentation method. The automatic prompting segmentation method achieves dynamic alignment between medical image features and segmentation tasks through cross-modal attention interaction between task tokens and output tokens, and generates segmentation mask images of each single modality by combining sparse prompting generation mechanism and dense prompting generation mechanism.

[0009] S2. The segmentation mask images of the three modalities are registered and feature extracted to obtain the structured medical descriptions of the segmentation mask images of each modality. The structured medical descriptions are then fused using a large language model to obtain the multimodal fused tumor segmentation mask image.

[0010] Specifically, step S1 includes the following steps:

[0011] S101, Token interaction initialization:

[0012] An initial task token is given by the medical image segmentation model. T t and output token T o The task token T t The output token is used to encode the global semantics of the segmentation task. T o Used to encode contextual information in historical segmentation results;

[0013] Task token T t With output token T o Perform cross-modal attention interaction and assign task tokens T t As a query from the output token T o Extract task-specific semantics and interactively generate new task tokens with fusion space-task alignment. and new output token ;

[0014] New task token and new output token By concatenating the elements along the dimension, we obtain the updated task token. The expression is as follows:

[0015] ;

[0016] in, For splicing operations; W p The projection matrix is ​​learnable. bp It is the bias vector;

[0017] MRI image features embedded with depth information are incorporated through a bidirectional cross-attention mechanism. F i With task tokens Feature-task alignment was performed to obtain enhanced MRI image features. and output token ;

[0018] S102, Sparse auto-suggestion generation:

[0019] Split output token ,forward n The vector represents the latest output token. ;back m Each vector represents the latest task token. To participate in the next round of learning and training;

[0020] Use coordinate decoder to output token Decode and generate sparse hints P s The spatial coordinates of sparse cue points are mapped to the image domain through an MLP network and sigmoid activation, as shown in the following expression:

[0021] ;

[0022] in, Indicates will n The vectors corresponding to n The coordinates of sparse cue points are aggregated into a coordinate set. It is the Sigmoid activation function. This indicates that for each output token Vectors undergo multilayer perceptron transformation, matrices W coord Used to map the coordinates of sparse cue points to the image space coordinate system;

[0023] S103, Dense Auto-Suggestion Generation:

[0024] Using a mask adapter of a lightweight CNN decoder, image features are... Convert to the logarithmic probability of the split P d It is used to predict the probability that each voxel in an image is a tumor.

[0025] Furthermore, in step S101, the bidirectional cross-attention mechanism includes:

[0026] Task to Image Attention:

[0027] Task token As a query, image features F i The purpose of using the key and value is to allow the task token to extract associated spatial features from the image, as shown in the following expression:

[0028] ;

[0029] in, Represents task token From image features F i The relevant features extracted from them will be As an output token ; This is the normalization function; W q , W k , W v The projection matrix is ​​learnable; d Normalization factor;

[0030] Image-to-Task Attention:

[0031] Image features F i As a query, the task token The purpose of using these as keys and values ​​is to allow image features to absorb global task semantics, as shown in the following expression:

[0032] ;

[0033] in, Representing image features F i According to the task token The semantically updated features will As an image feature .

[0034] Preferably, the medical image segmentation model integrates a self-ordering memory to enhance the segmentation mask image generated by automatic prompts, specifically including the following steps:

[0035] The MRI monomodal image to be segmented is decomposed along axial, coronal, sagittal, and reverse perspectives of the three views to obtain multi-view representations of anatomical structures:

[0036] ;

[0037] in, These represent the characteristics of axial, coronal, and sagittal MRI images in a single modality, as well as the reverse perspective of axial, coronal, and sagittal views, respectively.

[0038] The formula for multi-view fusion is:

[0039] ;

[0040] in, X fused This is the merged view after combining six views; w i Assign confidence weights to each view;

[0041] Based on the prediction results of the current unimodal image and the historical prediction results stored in the self-ordering memory, features with prediction probabilities higher than a set threshold are selected and retained. Then, redundant features similar to those in the self-ordering memory are removed from the current unimodal image using cosine similarity. The self-ordering memory is updated using the selected features from the current unimodal image, and the features in the updated self-ordering memory are weighted and aggregated to generate context-enhanced features. C ctx ;

[0042] Using the merged view as the key-value pair and the enhanced feature as the query, calculate the attention weight A:

[0043] ;

[0044] in, This is the normalization function; d Normalization factor;

[0045] The weighted features are mapped to the final unimodal segmentation result using a 3D CNN decoder:

[0046] ;

[0047] in, Y 3D This is a tumor segmentation mask image after single-modal enhancement of an MRI image; Conv 3 D For 3D CNN decoders, Sigmoid This is the activation function.

[0048] Furthermore, confidence weights for each view are calculated based on a self-ordering memory. w i :

[0049] ;

[0050] in, w i Representation of feature mapV i Confidence weights; V i Indicates the sixth view i Feature maps from multiple perspectives; Feature maps V i The features are the mean and standard deviation; MLP is a mapping function used to map the vector concatenated with the mean and standard deviation to a confidence energy; This is a normalization function used to normalize the confidence energy of the six views.

[0051] Specifically, step S2 includes the following steps:

[0052] S201, Multimodal Feature Quantification Analysis:

[0053] Medical image registration tools were used to register the mask images of T1c mode and DWI mode to the spatial coordinate system of T2 mode, ensuring that all mask images have the same voxel resolution and spatial position.

[0054] Feature extraction is performed on the registered masked image to generate a structured text description that can be processed by a large language model;

[0055] S202, using a large language model to generate fusion rules;

[0056] S203 uses fusion rules to fuse overlapping areas of the three modalities of the mask images and performs local optimization on non-overlapping conflict areas.

[0057] Furthermore, in step S201, the extracted features include:

[0058] Volumetric features: Calculate the tumor volume in each modal mask image;

[0059] Overlapping regions and overlap rates of tumor areas in three modal masked images;

[0060] Conflict regions in the tumor area in three modal mask images;

[0061] Morphological parameters: Compactness and eccentricity are calculated based on T2 modal mask images.

[0062] Further, step S202 includes:

[0063] Embed cervical cancer imaging knowledge into the input prompts of the large language model, including:

[0064] T1c enhancement area: represents a highly vascularized or active tumor, but may contain inflammation;

[0065] T2 high signal area: represents the main body of the tumor and possible edema, with good anatomical localization;

[0066] High signal area on DWI / low value area on ADC: Represents the dense active core of cells, with high sensitivity but may have T2 penetration artifacts;

[0067] Gold standard: The final tumor extent should integrate segmentation results from T1c / DWI and T2 modalities;

[0068] The fusion weights for each modality were set based on the characteristics of cervical cancer cases. The fusion weights for each modality are as follows:

[0069] ;in, w T1c , w T2 , w DWI The fusion weights are for T1c, T2, and DWI modalities, respectively. When the overlap rate of the tumor region in the masked images of the three modalities is lower than a set threshold, the fusion weight of the DWI modality is increased as follows: ,in, The fusion weights are updated after the DWI mode.

[0070] Furthermore, the overlapping regions of the tumor area in the three modal mask images are fused, and the fusion formula for the multimodal segmentation probability is as follows:

[0071]

[0072] in, Voxels representing the three modes respectively ( x , y , z The probability of it being diagnosed as a tumor; This represents the voxel after the fusion of the three modes ( x , y , z The probability of it being diagnosed as a tumor.

[0073] Furthermore, for conflict regions in the tumor area within the three modal mask images, the local optimization method is as follows:

[0074] The original images of the three modalities are cropped with the conflict area as the center to obtain local image patches;

[0075] Generate text prompts, including: conflict location, imaging features, conflict type, and clinical decision-making basis;

[0076] Local image patches and textual cues are input into a large VLM model for alignment analysis to generate decisions, confidence levels, and reasons.

[0077] For decision results with a confidence level greater than a set threshold, the fused multimodal segmentation probability map is updated; for decision results with a confidence level less than a set threshold, manual review is requested, and the fused multimodal segmentation probability map is updated based on the manual review results.

[0078] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0079] (1) This invention introduces a large language model to fuse the segmentation mask image of MRI multimodal images. It uses the powerful knowledge integration, reasoning and natural language interaction capabilities of the large language model to simulate expert decision-making logic. By combining the segmentation characteristics of different modalities, the fused tumor segmentation mask image is finally obtained, which greatly improves the segmentation accuracy of MRI images.

[0080] (2) This invention achieves dynamic alignment of medical image features and segmentation tasks through cross-modal attention interaction between task tokens and output tokens. Combined with sparse and dense cue generation mechanisms, it solves the problem of missegmentation caused by blurred boundaries of T2-weighted images, effectively suppresses magnetic susceptibility artifact interference, and significantly improves the accuracy and efficiency of single-modal segmentation of cervical cancer MRI images.

[0081] (3) This invention introduces a six-view decomposition and confidence-weighted fusion mechanism, combined with adaptive screening of historical prediction features, effectively integrates three-dimensional anatomical information, significantly suppresses the limitations of single-view imaging, improves the robustness of tumor spatial localization, and continuously optimizes the consistency of segmentation results by dynamically updating the memory bank.

[0082] (4) This invention embeds medical prior knowledge into a large language model to generate interpretable fusion rules and weight strategies, effectively coordinates multimodal conflicts, and adaptively adjusts weights through features such as quantified overlap rate to achieve complementarity between anatomical localization (T2) and functional information (DWI / T1c), ultimately generating fusion segmentation results that are more in line with clinical gold standards.

[0083] (5) This invention addresses conflict areas where multimodal coordination is difficult, by calling a large visual model for auxiliary analysis. This mechanism provides doctors with interventionable decision nodes, enhances the reliability of the system in complex cases, and reduces labeling inconsistencies caused by modal contradictions. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a flowchart illustrating an automatic segmentation method for cervical cancer MRI images based on multimodal fusion according to the present invention.

[0086] Figure 2 This is a schematic diagram of the architecture of the automatic prompt generation method in an embodiment of the present invention;

[0087] Figure 3 This is a schematic diagram of the architecture of the 3D medical image self-ordering memory update method based on multi-view fusion in an embodiment of the present invention;

[0088] Figure 4 This is a schematic diagram illustrating the process of fusing segmented mask images of three modes in an embodiment of the present invention;

[0089] Figure 5 This is a schematic diagram illustrating the process of optimizing local conflict areas in an embodiment of the present invention. Detailed Implementation

[0090] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0091] Reference Figure 1 This invention provides an automatic segmentation method for cervical cancer MRI images based on multimodal fusion, comprising the following steps:

[0092] S1. Obtain tumor segmentation mask images of cervical cancer MRI images in three modalities (T1c, T2, and DWI) through an automatic prompting segmentation method. The automatic prompting segmentation method achieves dynamic alignment between medical image features and segmentation tasks through cross-modal attention interaction between task tokens and output tokens, and generates segmentation mask images for each single modality by combining sparse prompting generation mechanisms and dense prompting generation mechanisms.

[0093] S2. The segmentation mask images of the three modalities are registered and feature extracted to obtain the structured medical descriptions of the segmentation mask images of each modality. The structured medical descriptions are then fused using a large language model to obtain the multimodal fused tumor segmentation mask image.

[0094] Specifically, step S1 includes the following steps:

[0095] S101, Token interaction initialization:

[0096] like Figure 2 As shown, an initial task token is given by the medical image segmentation model. Tt (Task token) and output token T o (Output token), the task token T t The output token is used to encode the global semantics of the segmentation task. T o Used to encode contextual information in historical segmentation results;

[0097] Task token T t With output token T o Perform cross-modal attention interaction and assign task tokens T t As a query from the output token T o Extract task-specific semantics and interactively generate new task tokens with fusion space-task alignment. and new output token ;

[0098] New task token and new output token By concatenating the elements along the dimension, we obtain the updated task token. The target token undergoes a nonlinear transformation through a two-layer multilayer perceptron (MLP) activated by the GELU function, followed by linear projection to maintain feature compactness, allowing the target token to absorb contextual cues specific to the instance. The expression for the concatenated new task token is as follows:

[0099] ;

[0100] in, For splicing operations; W p The projection matrix is ​​learnable. b p It is the bias vector;

[0101] MRI image features embedded with depth information are incorporated through a bidirectional cross-attention mechanism. F i With task tokens Feature-task alignment was performed to obtain enhanced MRI image features. and output token This achieves deep coupling between the target token and the image, preserving the task-related semantic context.

[0102] S102, Sparse auto-suggestion generation:

[0103] Split output token ,forwardn The vector represents the latest output token. ;back m Each vector represents the latest task token. To participate in the next round of learning and training;

[0104] Use coordinate decoder to output token Decode and generate sparse hints P s The spatial coordinates of sparse cue points are mapped to the image domain through an MLP network and sigmoid activation, as shown in the following expression:

[0105] ;

[0106] in, Indicates will n The vectors corresponding to n The coordinates of sparse cue points are aggregated into a coordinate set. It is the Sigmoid activation function. This indicates that for each output token Vectors undergo multilayer perceptron transformation, matrices W coord Used to map the coordinates of sparse cue points to the image space coordinate system;

[0107] S103, Dense Auto-Suggestion Generation:

[0108] Using a mask adapter of a lightweight CNN decoder, image features are... Convert to logarithmic probability P d It is used to predict the probability that each voxel in an image is a tumor.

[0109] Furthermore, in step S101, the bidirectional cross-attention mechanism includes:

[0110] Task to Image Attention:

[0111] Task token As a query, image features F i The purpose of using the key and value is to allow the task token to extract associated spatial features from the image, as shown in the following expression:

[0112] ;

[0113] in, Indicates task token From image features F i The relevant features extracted from them will be As an output token ; This is the normalization function; W q , W k , W v The projection matrix is ​​learnable; d Normalization factor;

[0114] Image-to-Task Attention:

[0115] Image features F i As a query, the task token The purpose of using these as keys and values ​​is to allow image features to absorb global task semantics, as shown in the following expression:

[0116] ;

[0117] in, Representing image features F i According to the task token The semantically updated features will As an image feature .

[0118] Preferably, the medical image segmentation model integrates a self-ordering memory to enhance the segmentation mask image generated by automatic prompts. This embodiment proposes a self-ordering memory update within the MedSAM-2 architecture. Given a 3D image, multi-view fusion is used to enhance the 3D feature representation of the cervical cancer image. Then, the image encoding with enhanced 3D feature expression is obtained through self-ordering memory update. The self-ordering memory update framework based on the automatically prompted 3D image enhanced by multi-view fusion is as follows: Figure 3 As shown; specifically including the following steps:

[0119] The MRI monomodal image to be segmented is decomposed along axial, coronal, sagittal, and reverse perspectives of the three views to obtain multi-view representations of anatomical structures:

[0120] ;

[0121] in, These represent the characteristics of axial, coronal, and sagittal MRI images in a single modality, as well as the reverse perspective of axial, coronal, and sagittal views, respectively.

[0122] The formula for multi-view fusion is:

[0123] ;

[0124] in, Xfused This is the merged view after combining six views; w i Assign confidence weights to each view;

[0125] Based on the prediction results of the current unimodal image and the historical prediction results stored in the self-ordering memory, features with prediction probabilities higher than a set threshold are selected and retained. Then, redundant features similar to those in the self-ordering memory are removed from the current unimodal image using cosine similarity. The self-ordering memory is updated using the selected features from the current unimodal image, and the features in the updated self-ordering memory are weighted and aggregated to generate context-enhanced features. C ctx ;

[0126] Using the merged view as the key-value pair and the enhanced feature as the query, calculate the attention weight A:

[0127] ;

[0128] in, This is the normalization function; d Normalization factor;

[0129] The weighted features are mapped to the final unimodal segmentation result using a 3D CNN decoder:

[0130] ;

[0131] in, Y 3D This is a tumor segmentation mask image after single-modal enhancement of an MRI image; Conv 3 D For 3D CNN decoders, Sigmoid This is the activation function.

[0132] Furthermore, confidence weights for each view are calculated based on a self-ordering memory. w i :

[0133] ;

[0134] in, w i Representation of feature map V i Confidence weights; V i Indicates the sixth view i Feature maps from multiple perspectives; Feature maps V iThe features are the mean and standard deviation; MLP is a mapping function used to map the vector concatenated with the mean and standard deviation to a confidence energy; This is a normalization function used to normalize the confidence energy of the six views.

[0135] Specifically, in step S2, the Large Language Model (LLM) dynamically adapts by adjusting fusion parameters in real time based on tumor volume, location, and morphological characteristics. Simultaneously, a hierarchical processing strategy is employed: globally (approximately 80% of the region), the LLM generates weights based on case characteristics; locally (approximately 20% of high-impact regions), the VLM model focuses on resolving key conflict areas. Based on natural language expressions of medical evidence, it supports physician review and enhances the clinical interpretability of each decision. For example... Figure 4 As shown, the specific steps include:

[0136] S201, Multimodal Feature Quantification Analysis:

[0137] Since there are spatial differences in the acquisition of MRI images of the three modalities of T1c, T2 and DWI, this embodiment uses a medical image registration tool to register the mask images of T1c modality and DWI modality to the spatial coordinate system of T2 modality, ensuring that all mask images have the same voxel resolution and spatial position, thereby maximizing mutual information.

[0138] Feature extraction is performed on the registered masked image to generate a structured text description that can be processed by a large language model;

[0139] S202, using Large Language Model (LLM) to generate fusion rules;

[0140] S203 uses fusion rules to fuse overlapping areas of the three modalities of the mask images and performs local optimization on non-overlapping conflict areas.

[0141] Further, in step S201, the segmentation results of the three modalities are converted into structured text descriptions, including the overall volume, location (anterior / posterior cervical lip, depth of invasion), and morphology (regular / irregular) of lesions in each modality; simultaneously, the volume and location of the overlapping regions (consensus regions) of the three modalities are expressed, as well as the volume, location, and confidence level of the unique regions of each modality. Key conflict regions are listed (e.g., coordinates (x,y,z): T1c=1, T2=0, DWI=1), and qualitative descriptions of the original images are provided (e.g., "lesion boundaries are blurred on T2", "lesions show significantly high signal on DWI"). Feature extraction is performed on the registered mask to generate structured descriptions for LLM decision-making. The extracted features include:

[0142] Volumetric features: Calculate the tumor volume in each modal mask image;

[0143] Overlapping regions and overlap rates of tumor areas in three modal masked images;

[0144] Conflicting regions of tumor areas in three modal masked images, such as T1c+DWI positive / T2 negative areas, or T2 positive / DWI negative areas;

[0145] Morphological parameters: Compactness and eccentricity were calculated based on T2 modal mask images;

[0146] The above features are encapsulated in JSON format and input into the LLM model.

[0147] Furthermore, in step S202, cervical cancer imaging characteristics are first embedded in the input prompts to provide medical knowledge prompts, and different weight benchmarks are assigned to the three modalities. For example, the active tumor specificity of the T1c enhancement area is set with a weight benchmark range of 0.8-1.2; the anatomical range of the T2 high signal area is good but the boundary is blurred, and the weight benchmark range is set with a weight benchmark range of 0.9-1.4; the active core sensitivity of the DWI high signal area is high, and the weight benchmark range is set with a weight benchmark range of 1.2-1.5.

[0148] Embed cervical cancer imaging knowledge into the input prompts of the large language model, including:

[0149] T1c enhancement area: represents a highly vascularized or active tumor, but may contain inflammation;

[0150] T2 high signal area: represents the main body of the tumor and possible edema, with good anatomical localization;

[0151] High signal area on DWI / low value area on ADC: Represents the dense active core of cells, with high sensitivity but may have T2 penetration artifacts;

[0152] Gold standard: The final tumor extent should integrate segmentation results from T1c / DWI and T2 modalities;

[0153] The fusion weights for each modality were set based on the characteristics of cervical cancer cases. The fusion weights for each modality are as follows:

[0154] ;in, w T1c , w T2 , w DWI The fusion weights are T1c, T2, and DWI modalities, respectively. When the overlap rate of the tumor region in the masked images of the three modalities is lower than a set threshold, the fusion weight of the DWI modality is increased as follows: ,in, The fusion weights are updated after the DWI mode.

[0155] Then, the doctor analyzes the image mask features of different modalities of cervical cancer images, combines them with LLM to generate conflict rules, and outputs programmable natural language rules, such as:

[0156] "If the DWI signal intensity is >120 and the distance from the T2 boundary is <2mm, then the T1c+DWI positive area is preserved."

[0157] Structured rules: "Consensus regions (3 positives) are directly retained; regions that are positive for T1c+DWI but negative for T2 are retained if the DWI signal intensity is greater than the threshold, otherwise they are discarded; regions with only a single modality of positivity require careful verification."

[0158] Dynamic weights: For example, “Global weights: DWI: 1.2, T2: 1.1, T1c: 0.8. For conflict areas located in the cervical stroma, increase the T2 weight to 1.3.”

[0159] Post-processing suggestions: For example, "Perform a 5mm closing operation on the fusion result to fill small holes and remove isolated regions with an area <50mm²". Finally, output the above programmable natural language rules in a structured format and force LLM to return in JSON format.

[0160] Furthermore, the overlapping regions of the tumor area in the three modal mask images are fused, and the fusion formula for the multimodal segmentation probability is as follows:

[0161]

[0162] in, Voxels representing the three modes respectively ( x , y , z The probability of it being diagnosed as a tumor; This represents the voxel after the fusion of the three modes ( x , y , z The probability of it being diagnosed as a tumor.

[0163] Furthermore, for conflict regions in the tumor area within the three modal mask images, the local optimization method is as follows:

[0164] The original images of the three modalities are cropped with the conflict area as the center to obtain local image patches;

[0165] Generate text prompts, including: conflict location, imaging features, conflict type, and clinical decision-making basis;

[0166] Local image patches and textual cues are input into a large VLM model for alignment analysis to generate decisions, confidence levels, and reasons.

[0167] For decision results with a confidence level greater than a set threshold, the fused multimodal segmentation probability map is updated; for decision results with a confidence level less than a set threshold, manual review is requested, and the fused multimodal segmentation probability map is updated based on the manual review results.

[0168] For conflict areas that cannot be directly merged, a Visual Model (VLM) is invoked to optimize and decide on local conflicts. The decision-making process is as follows: Figure 5 As shown, a 64×64 pixel patch of the original image is cropped centered on the conflict area of ​​T1c / T2 / DWI for region focusing. Then, a VLM (Vibration Mode and Light) cueing process is designed. Based on the image description (the location of T1c-enhanced, T2-weighted DWI, and the color of the standard conflict area), combined with medical decision-making criteria: if T1c+DWI positive / T2 negative: it may be a small active tumor. VLM further analyzes the DWI signal. If the DWI signal is greater than the threshold, the T1c+DWI positive area is retained; if T2 positive / DWI negative: it may be edema or a false positive. VLM further analyzes the anatomical location. If the T2 positive area invades the cervical stroma, the T2 positive area is retained. This process analyzes the image description based on medical decision-making criteria, performs final decision parsing, and returns a binary decision (0 / 1) and the medical analysis reason, thus obtaining the final fused expression formula.

[0169] This invention first uses a dynamic automatic prompting method to automatically segment the T1c, T2, and DWI modalities of cervical cancer MRI, obtaining individual segmentation masks. Then, the three modalities of cervical cancer MRI are fused using a medical knowledge prediction model to achieve more accurate fused features. Fusing the segmentation results (masks) of the three modalities of cervical cancer MRI into a more efficient and accurate segmentation mask is a key step in improving the robustness and clinical applicability of the model. The core idea is to utilize the complementary information provided by different modalities (T1c enhances blood supply / active areas, T2 shows anatomical structures / edema, and DWI shows cell density / diffusion-restricted areas) and reduce missegmentation, noise, or bias in individual modalities through a fusion strategy.

[0170] The fusion scheme of this invention significantly improves the segmentation effect of cervical cancer:

[0171] 1) Improved accuracy: Global dynamic weighted fusion (LLM generating weights) improves the Dice coefficient to 0.88±0.03, which is 12% higher than that of single-modality fusion.

[0172] 2) False positive suppression: VLM analysis of conflict areas reduces false positives due to edema, decreasing the false positive rate by 35% (especially in areas with blurred T2 boundaries).

[0173] 3) Anatomical-functional complementarity: T2 provides precise anatomical contours, DWI / T1c fusion locks in the active core, and the gold standard alignment error is <1.5mm;

[0174] 4) Resolution of discrepancies in labeling: The physician review acceptance rate was 92%, because the LLM rules provided interpretable medical evidence (such as "DWI weighting is increased because necrosis rate <10%").

[0175] 5) Enhanced robustness: Improved tolerance to magnetic susceptibility artifacts (DWI) and motion artifacts (T2), with segmentation stability increased by 40% compared to single-mode.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automatic segmentation method for cervical cancer MRI images based on multimodal fusion, characterized in that, Includes the following steps: S1. Obtain tumor segmentation mask images for three modalities (T1c, T2, and DWI) of cervical cancer MRI images using an automatic prompting segmentation method. This automatic prompting segmentation method achieves dynamic alignment between medical image features and the segmentation task through cross-modal attention interaction between task tokens and output tokens. It also generates segmentation mask images for each single modality by combining sparse and dense prompting generation mechanisms. Specifically, this includes the following steps: S101, Token interaction initialization: An initial task token is given by the medical image segmentation model. and output token The task token The output token is used to encode the global semantics of the segmentation task. Used to encode contextual information in historical segmentation results; Task token With output token Perform cross-modal attention interaction and assign task tokens As a query from the output token Extract task-specific semantics and interactively generate new task tokens with fusion space-task alignment. and new output token ; New task token and new output token By concatenating the elements along the dimension, we obtain the updated task token. The expression is as follows: ； in, For splicing operations; The projection matrix is ​​learnable. It is the bias vector; MRI image features embedded with depth information are incorporated through a bidirectional cross-attention mechanism. With task tokens Feature-task alignment was performed to obtain enhanced MRI image features. and output token ; S102, Sparse auto-suggestion generation: Split output token The first n vectors are the latest output tokens. The last m vectors are the latest task tokens. To participate in the next round of learning and training; Use coordinate decoder to output token Decode and generate sparse hints The spatial coordinates of sparse cue points are mapped to the image domain through an MLP network and sigmoid activation, as shown in the following expression: ； in, This means aggregating the coordinates of n sparse cue points corresponding to n vectors into a single coordinate set. It is the Sigmoid activation function. This indicates that for each output token Vectors undergo multilayer perceptron transformation, matrices Used to map the coordinates of sparse cue points to the image space coordinate system; S103, Dense Auto-Suggestion Generation: Using a mask adapter of a lightweight CNN decoder, image features are... Convert to the logarithmic probability of the split This is used to predict the probability that each voxel in an image is a tumor; S2. The segmentation mask images of the three modalities are registered and feature extracted to obtain the structured medical descriptions of the segmentation mask images of each modality. The structured medical descriptions are then fused using a large language model to obtain the multimodal fused tumor segmentation mask image.

2. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 1, characterized in that, In step S101, the bidirectional cross-attention mechanism includes: Task to Image Attention: Task token As a query, image features The purpose of using the key and value is to allow the task token to extract associated spatial features from the image, as shown in the following expression: ； in, Indicates task token From image features The relevant features extracted from them will be As an output token ; This is the normalization function; , , is the learnable projection matrix; d is the normalization factor; Image-to-Task Attention: Image features As a query, the task token The purpose of using these as keys and values ​​is to allow image features to absorb global task semantics, as shown in the following expression: ； in, Representing image features According to the task token The semantically updated features will As an image feature .

3. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 1, characterized in that, The medical image segmentation model integrates a self-ordering memory to enhance the segmentation mask image generated by automatic prompting, specifically including the following steps: The MRI monomodal image to be segmented is decomposed along axial, coronal, sagittal, and reverse perspectives of the three views to obtain multi-view representations of anatomical structures: ； in, These represent the characteristics of axial, coronal, and sagittal MRI images in a single modality, as well as the reverse perspective of axial, coronal, and sagittal views, respectively. The formula for multi-view fusion is: ； in, This is the merged view after combining six views; Assign confidence weights to each view; Based on the prediction results of the current unimodal image and the historical prediction results stored in the self-ordering memory, features with prediction probabilities higher than a set threshold are selected and retained. Then, redundant features similar to those in the self-ordering memory are removed from the current unimodal image using cosine similarity. The self-ordering memory is updated using the selected features from the current unimodal image, and the features in the updated self-ordering memory are weighted and aggregated to generate context-enhanced features. ; Using the merged view as the key-value pair and the enhanced feature as the query, calculate the attention weight A: ； in, d is the normalization function; d is the normalization factor; The weighted features are mapped to the final unimodal segmentation result using a 3D CNN decoder: ； in, This is a tumor segmentation mask image after single-modal enhancement of an MRI image; Conv3D is a 3D CNN decoder, and Sigmoid is the activation function.

4. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 3, characterized in that, Calculate the confidence weights of each view based on a self-ordering memory. : ； in, Representation of feature map Confidence weights; This represents the feature map of the i-th viewpoint in the sixth view; Feature maps The features are the mean and standard deviation; MLP is a mapping function used to map the vector concatenated with the mean and standard deviation to a confidence energy; This is a normalization function used to normalize the confidence energy of the six views.

5. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 1, characterized in that, Step S2 specifically includes the following steps: S201, Multimodal Feature Quantification Analysis: Medical image registration tools were used to register the mask images of T1c mode and DWI mode to the spatial coordinate system of T2 mode, ensuring that all mask images have the same voxel resolution and spatial position. Feature extraction is performed on the registered masked image to generate a structured text description that can be processed by a large language model; S202, using a large language model to generate fusion rules; S203 uses fusion rules to fuse overlapping areas of the three modalities of the mask images and performs local optimization on non-overlapping conflict areas.

6. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 5, characterized in that, In step S201, the extracted features include: Volumetric features: Calculate the tumor volume in each modal mask image; Overlapping regions and overlap rates of tumor areas in three modal masked images; Conflict regions in the tumor area in three modal mask images; Morphological parameters: Compactness and eccentricity are calculated based on T2 modal mask images.

7. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 5, characterized in that, Step S202 includes: Embedding cervical cancer imaging knowledge into the input prompts of a large language model, including: T1c enhancement area: represents a highly vascularized or active tumor, but may contain inflammation; T2 high signal area: represents the main body of the tumor and possible edema, with good anatomical localization; High signal area on DWI / low value area on ADC: Represents the dense active core of cells, with high sensitivity but may have T2 penetration artifacts; Gold standard: The final tumor extent should integrate the segmentation results of T1c modality / DWI modality and T2 modality; The fusion weights for each modality were set based on the characteristics of cervical cancer cases. The fusion weights for each modality are as follows: ;in, , , The fusion weights are for T1c, T2, and DWI modalities, respectively. When the overlap rate of the tumor region in the masked images of the three modalities is lower than a set threshold, the fusion weight of the DWI modality is increased as follows: ,in, The fusion weights are updated after the DWI mode.

8. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 7, characterized in that, In step S203, the overlapping regions of the tumor area in the three modal mask images are fused. The fusion formula for the multimodal segmentation probability is as follows: ； in, These represent the probabilities of identifying a voxel (x, y, z) in three different modalities as a tumor. This represents the probability that the voxel (x,y,z) after the fusion of the three modalities is identified as a tumor.

9. The automatic segmentation method for cervical cancer MRI images based on multimodal fusion as described in claim 5, characterized in that, For conflict regions in the tumor area within the three modal mask images, the local optimization method is as follows: The original images of the three modalities are cropped with the conflict area as the center to obtain local image patches; Generate text prompts, including: conflict location, imaging features, conflict type, and clinical decision-making basis; Local image patches and textual cues are input into a large VLM model for alignment analysis to generate decisions, confidence levels, and reasons. For decision results with a confidence level greater than a set threshold, the fused multimodal segmentation probability map is updated; for decision results with a confidence level less than a set threshold, manual review is requested, and the fused multimodal segmentation probability map is updated based on the manual review results.

Images