A multi-modal multi-target segmentation method and system for microvascular decompression

Abstract

The embodiment of the application relates to the technical field of image segmentation, in particular to a multi-modal multi-target segmentation method and system for microvascular decompression, which comprises the following steps: taking images of different modes as inputs of a multi-modal model, sequentially passing through a shared encoder and a specific decoder to obtain pixel-level segmentation results and projection features respectively; calculating a pixel-level segmentation loss function, and calculating a cross-modal mutual information distillation loss based on the pixel-level segmentation loss function; calculating a projection topology loss based on the projection features; storing the projection features in a storage library and calculating an inter-modal contrast loss; obtaining a loss function of the multi-modal model based on the cross-modal mutual information distillation loss, the projection topology loss and the inter-modal contrast loss; and performing multi-target segmentation based on the loss function of the multi-modal model. The multi-modal multi-target segmentation method for microvascular decompression realizes automatic and accurate segmentation of MVD-related structures.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image segmentation, and particularly relate to a multi-modal multi-target segmentation method and system for microvascular decompression. BACKGROUND

[0002] Microvascular decompression (MVD) is a widely used neurosurgical procedure for treating cranial nerve compression, used to treat symptoms such as pain or muscle spasm caused by intracranial vessels compressing nerves. Segmentation of MVD-related structures, including the brainstem, nerves, arteries, and veins, is crucial for preoperative planning and intraoperative decision-making.

[0003] In current clinical applications, doctors usually segment the brainstem, nerves, and arteries and veins related to MVD surgery using manual delineation, which is time-consuming and labor-intensive and relies on clinical experience. To quickly improve clinical efficiency, researchers have proposed automatic and semi-automatic methods to extract MVD-related structures. For example, in the clinic, three-dimensional visualization is performed with the help of 3D visualization software such as 3D slicer and BrainLab, but these methods rely on manual adjustment of various complex parameters to achieve better visualization, which is often limited by the experience of doctors, resulting in large human errors.

[0004] With the development of deep learning, existing technologies propose many deep neural network models for automatic segmentation of medical images. Most of the neural network models are based on improvements of U-Net, but in the segmentation of brainstem, nerves, and blood vessels related to MVD, existing segmentation methods only consider a single modality, ignoring the complementary information provided by two modalities and the resulting performance improvement. SUMMARY

[0005] Embodiments of the present application provide a multi-modal multi-target segmentation method and system for microvascular decompression, which realizes automatic and accurate segmentation of MVD-related structures.

[0006] To solve the above technical problems, in a first aspect, the embodiments of the present application provide a multi-modal multi-target segmentation method for microvascular decompression, comprising the following steps: first, taking images of different modalities as inputs of a multi-modal model, sequentially passing through a shared encoder and a specific decoder for feature extraction, to obtain pixel-level segmentation results and projection features respectively; then, based on the segmentation results, calculating a pixel-level segmentation loss function, and based on the pixel-level segmentation loss function, calculating a cross-modal mutual information distillation loss; next, based on the projection features, calculating a projection topology loss; storing the projection features in a storage library, and based on the projection features, calculating an inter-modal contrast loss; finally, based on the cross-modal mutual information distillation loss, the projection topology loss and the inter-modal contrast loss, obtaining a loss function of the multi-modal model; based on the loss function of the multi-modal model, performing multi-target segmentation.

[0007] In some example embodiments, taking images of different modalities as inputs of a multi-modal model, sequentially passing through a shared encoder and a specific decoder for feature extraction, to obtain pixel-level segmentation results and projection features respectively, comprises: taking images of different modalities as inputs of a multi-modal model, concatenating multiple images of different modalities along the channel after inputting the shared encoder; after the multiple images of different modalities are processed by the shared encoder, the images are split along the channel and input into the specific decoder corresponding to the modality of the image respectively, and after feature extraction by the specific decoder, pixel-level segmentation results and projection features are output respectively.

[0008] In some example embodiments, the images of different modalities include images of T2 and TOF modalities; the specific decoder includes a T2 specific decoder corresponding to the T2 modality and a TOF specific decoder corresponding to the TOF modality.

[0009] In some example embodiments, given the prediction probability of T2 and TOF modalities with respect to the c-th class ∈ {brain stem, artery} and The expression of the cross-modal mutual information distillation loss is:

[0010]

[0011] wherein, L IMD is the cross-modal mutual information distillation loss, [·,·] represents the concatenation operation, D KL represents the calculation of the KL divergence.

[0012] In some example embodiments, based on the projected features, the inter-modal contrast loss is calculated, including: inputting the feature vector from a specific decoder into a projection head; grouping the projection head according to the semantic categories in the label to obtain a projection vector, and storing it in a memory library; inputting the projection vector into a prediction head; constructing a contrast loss, and adopting a cosine similarity to measure the difference between different categories, to calculate the inter-modal contrast loss.

[0013] In some example embodiments, the expression of the inter-modal contrast loss is:

[0014]

[0015] wherein, L contr is the inter-modal contrast loss, N c =4 represents the number of all modalities, f c and z c represent the projection vector and the prediction vector of the category c respectively, and · represents dot product.

[0016] In some example embodiments, based on the projected features, the projection topology loss is calculated, including: assuming that PD p and PD g are the persistent graphs of the prediction result and the label respectively, and adopting the Wasserstein distance to find the best match between the persistent graphs of the prediction result and the label; based on the best match, calculating the squared distance between each pair of matched points PD p and PD g ; evaluating the topological features on the two-dimensional projection of the three-dimensional prediction and the true value to reduce the computational complexity; defining the minimum intensity projection of the prediction as the lowest intensity along a given axis, to calculate the projection topology loss.

[0017] In some example embodiments, the squared distance between each pair of matched points PD p and PD g is calculated, and the expression is:

[0018]

[0019] wherein, u∈PD p , v∈PD g , and W1 represents the first-order Wasserstein distance;

[0020] The expression of the projection topology loss is:

[0021]

[0022] wherein, L topo is the projection topology loss; is the projection on a given axis.

[0023] In some example embodiments, the loss function of the multi-modal model is represented as:

[0024] L = L seg + λ1L IMD + λ2L contr + λ3L topo

[0025] wherein L seg is a dice loss and cross-entropy loss equally weighted sum, L IMD is a cross-modal mutual information distillation loss, L contr is an inter-modal contrastive loss, L topo is a projection topology loss, and λ1, λ2, λ3 are hyperparameters that adjust the loss values.

[0026] In a second aspect, the embodiments of the present application also provide a multi-modal multi-target segmentation system for microvascular decompression, comprising: a feature extraction module, a knowledge distillation module, a class contrast module, a topology preservation constraint module and a data processing module connected in sequence; wherein the feature extraction module is configured to take images of different modalities as inputs of a multi-modal model, and sequentially pass through a shared encoder and a specific decoder for feature extraction, to obtain pixel-level segmentation results and projection features respectively, and store the projection features in a storage library; the knowledge distillation module is configured to calculate a pixel-level segmentation loss function according to the segmentation results, and calculate a cross-modal mutual information distillation loss based on the pixel-level segmentation loss function; the class contrast module comprises a contrast learning module with a memory bank, and is configured to calculate an inter-modal contrastive loss through the contrast learning module according to the projection features; the topology preservation constraint module is configured to calculate a projection topology loss according to the projection features; the data processing module is configured to obtain a loss function of the multi-modal model according to the cross-modal mutual information distillation loss, the projection topology loss and the inter-modal contrastive loss, and perform multi-target segmentation based on the loss function of the multi-modal model.

[0027] The technical solutions provided by the embodiments of the present application have at least the following advantages:

[0028] The embodiment of the application provides a multi-modal multi-target segmentation method and system for microvascular decompression, and the method comprises the following steps: firstly, taking images of different modes as the input of a multi-modal model, sequentially performing feature extraction through a shared encoder and a specific decoder, and respectively obtaining a pixel-level segmentation result and projection features; then, based on the segmentation result, a pixel-level segmentation loss function is calculated, and based on the pixel-level segmentation loss function, a cross-modal mutual information distillation loss is calculated; next, based on the projection features, a projection topology loss is calculated; the projection features are stored in a storage library, and based on the projection features, an inter-modal contrast loss is calculated; finally, based on the cross-modal mutual information distillation loss, the projection topology loss and the inter-modal contrast loss, a loss function of the multi-modal model is obtained; and based on the loss function of the multi-modal model, multi-target segmentation is performed.

[0029] The application provides a multi-modal multi-target segmentation method for microvascular decompression, which is a multi-modal method for key structure segmentation related to MVD surgery based on knowledge-driven mutual distillation and topological constraint. In order to utilize the advantage knowledge of each mode, the application proposes a clinical knowledge guided mutual distillation scheme. The mutual distillation guided by clinical knowledge helps to distill the cross-modal artery knowledge, so that the performance of the artery is maximized. In view of the false recognition caused by fuzzy categories, the application proposes an inter-class contrast learning module to expand the distance between categories in the potential space, so as to obtain more discriminative feature representation. A category-based contrast loss method is proposed to construct a stronger feature space and improve the inter-class separability. In order to enhance the remote structure information, the application also proposes a projection topology loss based on persistent homology, which forces the model to learn topologically consistent features. A large number of experiments are performed on the MRI data set using the multi-modal model, and the experimental results are compared with other deep learning-based segmentation methods. The results show the effectiveness of the method. The method also provides a new paradigm for segmentation of key structures related to MVD. BRIEF DESCRIPTION OF DRAWINGS

[0030] One or more embodiments are illustrated by way of example in the accompanying drawings that are not intended to be limiting of the application, unless otherwise specifically indicated, the drawing figures are not necessarily to scale.

[0031] Figure 1 A flowchart of a multi-modal multi-target segmentation method for microvascular decompression provided by an embodiment of the application;

[0032] Figure 2 A structure diagram of a multi-modal multi-target segmentation system for microvascular decompression provided by an embodiment of the application;

[0033] Figure 3is a specific flowchart of a multi-modal multi-target segmentation method for microvascular decompression provided by an embodiment of the present application;

[0034] Figure 4 is a flowchart of maintaining the continuity of blood vessels and nerves in a topology maintaining constraint module provided by an embodiment of the present application;

[0035] Figure 5 is a segmentation result visualization schematic diagram of each method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0036] As can be known from the background, the existing segmentation method has the technical problem of only considering a single modality, ignoring the complementary information provided by the two modalities and the performance improvement following the complementary information.

[0037] Due to the limited information provided by a single modality and the complex topology of blood vessels and nerves, automatic segmentation of structures related to MVD is still a challenge for current methods. MVD is the most effective surgical treatment for trigeminal neuralgia, hemifacial spasm and glossopharyngeal neuralgia. The process of the surgery is to open the skull to reach the lateral brainstem after the sacrum, and then prevent polytetrafluoroethylene between the nerve and the culprit blood vessel. Therefore, accurate segmentation of key structures such as brain tissue, cranial nerves and surrounding blood vessels before surgery is crucial for preoperative assessment and intraoperative decision-making. In clinical practice, neurosurgeons use high-resolution T2-weighted imaging (HR-T2WI) to comprehensively assess anatomical structures, and use three-dimensional time-of-flight magnetic resonance angiography (3D-TOF) to assist arterial examination to find potential sites of nerve and blood vessel compression. HR-T2WI provides fine and detailed imaging, which can clearly distinguish blood vessels in low signal intensity brain, nerves and high signal intensity cerebrospinal fluid (CSF). Its excellent spatial resolution makes the anatomical analysis of different structures have good contrast. In addition, 3D-TOF provides clear visualization of arteries, which is in sharp contrast to cerebrospinal fluid. However, in HR-T2WI, it is challenging to distinguish arteries, veins and nerves due to their similar signal intensities, while in 3D-TOF, it is more difficult to segment these two structures due to the medium signal intensity of nerves and veins. Due to the different recognition abilities of the two modalities for different targets, the use of the two modalities can provide comprehensive anatomical details of key structures related to MVD, especially the spatial relationship between nerves and blood vessels, which is very helpful for accurate segmentation.

[0038] At present, there are limited methods for segmenting key structures related to MVD, and methods that can be expanded to this scenario also face some technical difficulties and limitations, mainly including:

[0039] (1) The existing deep learning-based method for MVD structure segmentation only considers the information provided by a single modality, ignoring the complementary information and mutual promotion that the two modalities can provide.

[0040] (2) The existing multi-modal method assigns the same weight to different modalities and relies on the learnable fusion of the deep learning model to equally consider each modality and the target. However, in the task of the present application, 3D-TOF only provides information related to the artery, and the fusion of the two modalities can improve the performance of artery segmentation, but this synergy may cause the performance of other targets to decrease.

[0041] (3) The current mainstream pixel-level loss function only considers the predicted value and the true value of a single pixel, ignoring the context information and anatomical structure. Such local properties cause these loss functions to be unable to capture structure and shape information; and the existing topology loss has the problem of too high computational complexity, resulting in too long operation time.

[0042] To solve the above technical problems, the embodiment of the present application provides a multi-modal multi-target segmentation method for microvascular decompression, which comprises the following steps: first, taking images of different modalities as inputs of a multi-modal model, sequentially passing through a shared encoder and a specific decoder for feature extraction, and obtaining pixel-level segmentation results and projection features, respectively; then, based on the segmentation results, a pixel-level segmentation loss function is calculated, and based on the pixel-level segmentation loss function, a cross-modal mutual information distillation loss is calculated; next, based on the projection features, a projection topology loss is calculated; the projection features are stored in a storage library, and based on the projection features, an inter-modal contrast loss is calculated; finally, based on the cross-modal mutual information distillation loss, the projection topology loss and the inter-modal contrast loss, a loss function of the multi-modal model is obtained; based on the loss function of the multi-modal model, multi-target segmentation is performed. The embodiment of the present application provides a multi-modal multi-target segmentation method and system for microvascular decompression, aiming to solve the problems of insufficient information provided by a single modality, insufficient consideration of clinical knowledge in multi-modal fusion, and complex vascular and neural topology structure and similar spatial structure in the process of key structure segmentation of a deep neural network model, so as to realize accurate segmentation of key structures.

[0043] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, the technical solutions claimed by the present application can be realized even without these technical details and various changes and modifications based on the following embodiments.

[0044] Reference is made to Figure 1The embodiment of the application provides a multi-modal multi-target segmentation method for microvascular decompression, and comprises the following steps:

[0045] In step S1, images of different modes are taken as inputs of a multi-modal model, and are sequentially subjected to feature extraction through a shared encoder and a specific decoder to obtain pixel-level segmentation results and projection features.

[0046] In step S2, a pixel-level segmentation loss function is calculated based on the segmentation results, and a cross-modal mutual information distillation loss is calculated based on the pixel-level segmentation loss function.

[0047] In step S3, a projection topological loss is calculated based on the projection features, the projection features are stored in a storage library, and an inter-modal contrast loss is calculated based on the projection features.

[0048] In step S4, a multi-modal model loss function is obtained based on the cross-modal mutual information distillation loss, the projection topological loss and the inter-modal contrast loss.

[0049] In step S5, multi-target segmentation is performed based on the multi-modal model loss function.

[0050] The application aims at the technical problem that the existing segmentation method only considers a single mode and ignores the complementary information provided by two modes and the performance improvement caused thereby, and proposes a multi-modal method for MVD surgery related key structure segmentation based on knowledge driven mutual distillation and topological constraint. In order to utilize the advantage knowledge of each mode, the application proposes a clinical knowledge guided mutual distillation scheme, calculates a cross-modal mutual information distillation loss based on the pixel-level segmentation loss function, and moreover, in view of the error recognition caused by fuzzy categories, proposes an inter-class contrast learning module to expand the distance between categories in a potential space, calculates an inter-modal contrast loss, so as to obtain a more discriminative feature representation, and in order to enhance remote structure information, proposes a projection topological loss based on persistent homology. Finally, a multi-modal model loss function is obtained based on the cross-modal mutual information distillation loss, the projection topological loss and the inter-modal contrast loss, and multi-target segmentation is performed based on the multi-modal model loss function.

[0051] Referring to Figure 2The embodiment of the application also provides a multi-modal multi-target segmentation system for microvascular decompression, comprising: a feature extraction module 101, a knowledge distillation module 102, a class contrast module 103, a topology preservation constraint module 104 and a data processing module 105 connected in sequence; wherein the feature extraction module 101 is used for taking images of different modalities as inputs of a multi-modal model, sequentially passing through a shared encoder and a specific decoder for feature extraction, respectively obtaining a pixel-level segmentation result and a projection feature, and storing the projection feature in a memory bank; the knowledge distillation module 102 is used for calculating a pixel-level segmentation loss function according to the segmentation result, and calculating a cross-modal mutual information distillation loss based on the pixel-level segmentation loss function; the class contrast module 103 comprises a contrast learning module with a memory bank, and calculates an inter-modal contrast loss through the contrast learning module according to the projection feature; the topology preservation constraint module 104 is used for calculating a projection topology loss according to the projection feature; and the data processing module 105 is used for obtaining a loss function of the multi-modal model according to the cross-modal mutual information distillation loss, the projection topology loss and the inter-modal contrast loss, and performing multi-target segmentation based on the loss function of the multi-modal model.

[0052] The multi-modal multi-target segmentation method for microvascular decompression of the application will be described in detail below. The technical scheme of the application is aimed at the segmentation of key structures related to MVD surgery, as shown in the following formula (1): Figure 3 As shown in the formula (1), the overall framework of the scheme comprises a shared encoder and two specific decoders. After the images of the T2 and TOF modalities are spliced along the channel, the images are input into the shared encoder, then split along the channel and input into the specific decoders, and the two specific decoders output the pixel-level segmentation result and the projection, respectively. The segmentation result is used for calculating the pixel-level segmentation loss function and the projection topology loss with the label, and the activation value before the softmax (normalized exponential function) is used for calculating the cross-modal mutual information distillation, and the projection is stored in the memory bank and used for calculating the inter-modal contrast loss. Figure 3 In the formula (1), Shared Encoder is the shared encoder, HRT2 Decoder is the T2 specific decoder, TOF Decoder is the TOF specific decoder, Segmentation head is the segmentation head, T2 Projection head is the T2 projection head, T2 Prediction head is the prediction head, and T2 Confidence-based selection and TOF Confidence-based selection are the T2 and TOF confidence-based selections, respectively. The application calculates the prediction probability of the T2 and TOF modalities with respect to the c-th class ∈ {brainstem, artery} by giving the prediction probability of the T2 and TOF modalities with respect to the c-th class ∈ {brainstem, artery} and The cross-modal mutual information distillation loss L is calculated IMD .

[0053] In some embodiments, in step S1, images of different modalities are taken as inputs of the multi-modal model, and sequentially pass through a shared encoder and a specific decoder for feature extraction, to obtain pixel-level segmentation results and projection features, including:

[0054] In step S101, images of different modalities are taken as inputs of the multi-modal model, and the multiple images of different modalities are spliced along the channel and input into the shared encoder.

[0055] In step S102, after the multiple images of different modalities are processed by the shared encoder, they are split along the channel and input into specific decoders corresponding to the modalities of the images, respectively. After feature extraction by the specific decoders, pixel-level segmentation results and projection features are output, respectively.

[0056] In some embodiments, the images of different modalities include images of T2 and TOF modalities; and the specific decoders include a T2 specific decoder corresponding to the T2 modality and a TOF specific decoder corresponding to the TOF modality.

[0057] The cross-modal mutual information distillation in the method utilizes the knowledge of the two modalities on the arteries, and designs a knowledge distillation loss based on confidence. First, the feature quality selection is performed, and the features that lead to accurate prediction when applied to the prediction layer are retained, that is, the pixels with a confidence higher than a threshold value φ are selected from the prediction probability map. Then, considering that the distillation process only includes the prediction probability of the brainstem and the arteries, it is proposed to decompose the multi-classification task into a binary classification problem, and minimize the difference between the prediction probabilities of the two decoders.

[0058] In some embodiments, given the prediction probabilities of the T2 and TOF modalities with respect to the c-th class ∈ {brainstem, artery} and The expression of the cross-modal mutual information distillation loss is:

[0059]

[0060] Wherein, L IMD is the cross-modal mutual information distillation loss, [·, ·] represents the splicing operation, D KL represents the calculation of the KL divergence.

[0061] In some embodiments, in step S3, based on the projection features, the inter-modality contrast loss is calculated, including:

[0062] In step S301, the feature vector from the specific decoder is input into the projection head.

[0063] Step S302, grouping the projection head according to the semantic category in the label, obtaining a projection vector, and storing in the memory library.

[0064] Step S303, inputting the projection vector to the prediction head.

[0065] Step S304, constructing a contrast loss, and using cosine similarity to measure the difference between different categories, calculating the inter-modal contrast loss.

[0066] In the category contrast module, the present application constructs a contrast learning module with a memory library. Here, the present application takes the T2 modality as an example. In the first stage of training, the feature vector P T2 is input to the projection head f T2 = G T2 (P T2 ). Then, f T2 is grouped according to the semantic category in the label y, and stored in the memory library. In the second stage of training, the projection vector f T2 is input to the prediction head z T2 = Q T2 (f T2 ).

[0067] The addition of the contrast loss is to enhance the similarity between the newly obtained prediction vector and the projection vector from the storage library. Since the memory library is filled with high-quality feature vectors after the first stage of training, the constructed contrast loss can help to pull the representations of the same category closer and pull the representations of different categories farther apart, thereby constructing better distinctions between different categories in the feature space. It is worth noting that the present application only extracts features of two categories, i.e., the brainstem and the artery, from the TOF modality. Cosine similarity is used to measure the difference between the category-specific representations, and forms the inter-modal contrast loss (category contrast loss) of the present application.

[0068] In some embodiments, the expression of the inter-modal contrast loss is:

[0069]

[0070] where L contr is the inter-modal contrast loss, N c = 4 represents the number of all modalities, f c and z c represent the projection vector and the prediction vector of category c, respectively, and · represents dot product.

[0071] In the topology preservation constraint module, the present application proposes to use persistent homology (PH) to maintain the continuity of blood vessels and nerves, and further proposes a projection topology loss to reduce the high computational complexity by an order of magnitude, such asFigure 4 is given an n-dimensional voxel block shaped as n1x n2x... x n d , which is represented as a cubical complex. PH tracks the evolution of topological features by a sequence of nested cubes and records it as a persistent diagram (PD). By assuming PD p and PD g are the predicted result and the label persistent diagram respectively, the Wasserstein distance is used to find the best match between the predicted result and the label persistent diagram; and the squared distance between each pair of matched points PD p and PD g is calculated, and the projection topological loss L topo is calculated.

[0072] In some embodiments, based on the projection feature, the projection topological loss is calculated, including: assuming PD p and PD g are the predicted result and the label persistent diagram respectively, the Wasserstein distance is used to find the best match between the predicted result and the label persistent diagram; based on the best match, the squared distance between each pair of matched points PD p and PD g is calculated; the topological feature is evaluated on the two-dimensional projection of the three-dimensional prediction and the true value to reduce the computational complexity; the minimum intensity projection of the prediction is defined as the lowest intensity along a given axis, and the projection topological loss is calculated.

[0073] In some embodiments, the squared distance between each pair of matched points PD p and PD g is calculated, and the expression is:

[0074]

[0075] where u PD p , v PD g , and W1 represents the first-order Wasserstein distance.

[0076] Using the true value and the predicted persistent diagram, the present application can quantify the topological features in the data and analyze the persistence at different resolutions. The above calculation process has the disadvantage of being very time-consuming, and the present application specifically proposes to evaluate the topological feature on the two-dimensional projection of the three-dimensional prediction and the true value, thereby reducing the computational complexity. The minimum intensity projection of the prediction is defined as the lowest intensity along a given axis, and the expression of the projection topological loss is:

[0077]

[0078] where L topois a projection topological loss; is a projection on a given axis.

[0079] In some embodiments, the loss function of the multi-modal model is represented as:

[0080] L=L seg +λ1L IMD +λ2L contr +λ3L topo

[0081] wherein L seg is a dice loss and cross-entropy loss equally weighted sum, L IMD is a cross-modal mutual information distillation loss, L contr is an inter-modal contrastive loss, L topo is a projection topological loss, λ1, λ2, λ3 are hyperparameters to adjust the loss value.

[0082] The scheme aims to realize the automatic and accurate segmentation of MVD related structures, and the present application constructs a multi-modal framework composed of a shared encoder and two specific decoders. The scheme can constrain the segmentation result by utilizing the mutual relationship and topology between different modal MRI images, thereby improving the segmentation accuracy and robustness. At the same time, the method can also improve the segmentation accuracy through mutual distillation, thereby further improving the segmentation effect.

[0083] Compared with the prior art, the multi-modal multi-target segmentation method for microvascular decompression provided by the present application has the advantages that: the present application proposes a U-Net network based on a hierarchical attention mechanism for segmenting the above two anatomical structures to help target positioning in STN-DBS surgery in view of the technical difficulties of automatically segmenting the subthalamic nucleus (STN) and the red nucleus, relative to the current manual target positioning, the present application provides an automatic target positioning method, which can be used to improve the efficiency of the operation; relative to the method based on machine learning and brain atlas, the present application utilizes the neural network to realize automatic feature extraction of the image, avoids the complex feature extraction process, and avoids the error caused by registering the brain atlas, can more quickly and accurately realize the accurate segmentation of the STN and the red nucleus, and reduce the dependence on expert experience; relative to the existing neural network segmentation method for the STN and the red nucleus, the present application can be targeted to the hierarchical feature extraction characteristics of the neural network, and process the features of different levels (low-level features and high-level semantic features) in a targeted manner, utilize the attention gate mechanism and the Transformer structure based on self-attention to improve the extraction efficiency of the low-level features and the high-level features in the neural network model, more efficiently mine the local features and the global context information of the neural network model, greatly improve the segmentation accuracy of the STN and the red nucleus, and are more conducive to the target positioning measurement of the STN.

[0084] The present application proposes a multi-modal segmentation method based on knowledge-driven mutual distillation and topological constraints for segmenting brainstem, nerves, arteries and veins related to MVD surgery, so as to help doctors save time in preoperative examination and planning and improve surgical efficiency. Compared with relying on three-dimensional visualization software assisted segmentation, the present application realizes complete automation and is not limited by the experience of doctors. Moreover, compared with the existing neural network segmentation method for MVD related key structures, the present application considers two modalities usually combined by doctors in clinical practice, and proposes bidirectional knowledge distillation of the two modalities combined with clinical knowledge. A contrast learning module with a memory bank is used to obtain more discriminative feature representations to solve the error recognition caused by class ambiguity. A persistent homology based projection topological loss is used to enhance the remote structure information and topological constraints, while avoiding excessive increase in computational complexity. The scheme of the present application can greatly improve the segmentation accuracy of MVD related key structures and greatly reduce the time of three-dimensional reconstruction in preoperative planning of doctors.

[0085] In order to verify the beneficial effects and feasibility of the multi-modal multi-target segmentation method provided by the present application, the method of the present application is experimentally verified, simulated and used to prove its feasibility. The following is a specific description of the experiment:

[0086] (1) Dataset

[0087] In this experiment, registered images of two modalities from 100 patients were collected, which were T2 weighted modality and TOF-MRA images respectively. The size of each slice was from 164*320 to 960*908, the layer thickness was from 0.2mm to 0.5mm, and the number of slices was from 72 to 80. The brainstem, nerves, arteries and veins were manually delineated by a neurosurgeon with 5 years of experience. In this experiment, 100 MRI samples and labels were used, of which 80 were used as a training set and the remaining 20 were used as a test set. Five-fold cross-validation was performed on the training set, and the segmentation results on the test set were obtained using the model of each validation. The performance of the method was evaluated using the evaluation results of the test set.

[0088] (2) Training process

[0089] All images were resampled to the same spatial resolution and cropped to [64, 128, 256] as model input before training. The data augmentation used in the present application includes random rotation, elastic deformation, Gaussian noise, mirror conversion and scaling. In the training phase, the present application uses the SGD optimizer, the learning rate is set to 0.01, the momentum is set to 0.99, and the weight decay is set to 3e-5.

[0090] The method is implemented using Python, based on the PyTorch 2.0.0 framework, and the model of the application is trained and tested on an NVIDIA GeForce GTX 4090 GPU. The training batch size is set to 2, and all models are trained based on the nnUNetv2 framework for 200 iterations, with 250 batches per iteration.

[0091] (3) Results show

[0092] The results obtained by the method are compared with U-Net, ASNet, Swin-UNetR, Edge-Reinforced network, CKD-TransBTS and H-DenseFormer, and the results obtained on 20 test samples are compared.

[0093] The indicators for comparison include Dice coefficient, clDice coefficient and 95% Hausdorff distance (HD95), where the smaller the HD95 indicator is, the better, and the other two indicators are better.

[0094] The results are shown in Table 1, and the method of the application is better than other methods in most indicators. Specifically, the precision of the method of the application on brainstem, artery, vein and nerve is 90.9%, 76.8%, 56.6% and 71.8% respectively. Compared with the widely used nnUNet, the model of the application achieves better results in segmentation. For example, the method of the application improves the Dice coefficient by 1.9%, 3.3% and 0.5% on artery, vein and nerve respectively. Such improvement shows that the method proposed in the application has better learning ability and generalization ability in MVD-related blood vessel and nerve segmentation tasks.

[0095] Some of the segmentation results are shown in Figure 5 , and the key and difficult areas of segmentation are marked by yellow boxes, and are corresponding to the intraoperative images. The visualization results show that the method of the application has higher consistency with manual segmentation annotation, indicating the effectiveness of the method of the application.

[0096] Table 1 Experimental results of different methods

[0097]

[0098] In Table 1, B, A, V and N represent brainstem, artery, vein and nerve respectively.

[0099] In addition, the application also conducts ablation experiments on the three loss functions added in the model to explore their influence on the experimental results. The experimental results are shown in Table 2. When any one of the loss functions is removed from the model of the application, the performance of the model will decrease, as shown in Table 2.

[0100] Table 2 ablation experiment results

[0101]

[0102] In Table 2, B, A, V, N respectively represent the brainstem, artery, vein and nerve.

[0103] According to the technical scheme, the application provides a multi-modal multi-target segmentation method and system for microvascular decompression, which comprises the following steps: first, taking images of different modalities as inputs of a multi-modal model, and sequentially performing feature extraction through a shared encoder and a specific decoder to obtain pixel-level segmentation results and projection features; then, calculating a pixel-level segmentation loss function based on the segmentation results, and calculating a cross-modal mutual information distillation loss based on the pixel-level segmentation loss function; next, calculating a projection topology loss based on the projection features; storing the projection features in a storage library, and calculating an inter-modality contrast loss based on the projection features; finally, obtaining a loss function of the multi-modal model based on the cross-modal mutual information distillation loss, the projection topology loss and the inter-modality contrast loss; and performing multi-target segmentation based on the loss function of the multi-modal model.

[0104] The application provides a multi-modal multi-target segmentation method for microvascular decompression, which is a multi-modal method for key structure segmentation related to MVD surgery based on knowledge-driven mutual distillation and topological constraints. In order to utilize the advantages of each modality, the application proposes a clinical knowledge-guided mutual distillation scheme. The mutual distillation guided by clinical knowledge helps to distill cross-modal artery knowledge and maximize the performance of the artery. In order to address the error recognition caused by fuzzy categories, the application proposes an inter-class contrast learning module to expand the distance between categories in the latent space, thereby obtaining more discriminative feature representations. A class-based contrast loss method is proposed to construct a more powerful feature space and improve inter-class separability. In order to enhance the remote structure information, the application also proposes a persistent homology-based projection topology loss to force the model to learn topologically consistent features. A large number of experiments are performed on the MRI dataset using the multi-modal model, and the experimental results are compared with other deep learning-based segmentation methods. The results demonstrate the effectiveness of the method. The method also provides a new paradigm for segmentation of key structures related to MVD.

[0105] Those skilled in the art can understand that the above embodiments are specific examples of the application, and in actual application, various changes can be made in form and details without departing from the spirit and scope of the application. Any person skilled in the art can make changes and modifications without departing from the spirit and scope of the application, therefore, the protection scope of the application should be limited by the scope defined in the claims.

Claims

1. A multi-modal multi-object segmentation method for microvascular decompression, characterized in that, The method comprises the following steps: taking images of different modalities as inputs of a multi-modal model, sequentially passing through a shared encoder and a specific decoder for feature extraction, and obtaining pixel-level segmentation results and projection features respectively; based on the segmentation results, calculating a pixel-level segmentation loss function, and based on the pixel-level segmentation loss function, calculating a cross-modality mutual information distillation loss; based on the projection features, calculating a projection topology loss; storing the projection features in a storage library, and based on the projection features, calculating a class contrast loss; based on the cross-modality mutual information distillation loss, the projection topology loss, and the class contrast loss, obtaining a loss function of the multi-modal model; based on the loss function of the multi-modal model, performing multi-object segmentation; Given T2 and TOF modalities, the predicted probability of a class c ∈ {brainstem, artery} and The representation of the cross-modality mutual information distillation loss is: wherein, is the cross-modal mutual information distillation loss, [·, ·] denotes the concatenation operation, denotes the computation of the KL divergence; based on the projection features, calculating a class contrast loss, comprising: inputting a feature vector from a specific decoder into a projection head to obtain a projection vector; grouping the projection vector according to semantic classes in a label, and storing it in a memory library; inputting the projection vector into a prediction head; constructing a contrast loss, and adopting cosine similarity to measure the difference between class-specific representations to form a class contrast loss; the expression of the class contrast loss is: wherein, is the class contrast loss, denotes the number of all classes, and denote the projection vector and the prediction vector for class c, respectively, and • denotes the dot product. based on the projection features, calculating a projection topology loss, comprising: Assume and are persistence diagrams of the prediction and the label, respectively, and the Wasserstein distance is used to find the best match between the persistence diagrams of the prediction and the label. Based on the best match, compute the squared distance between each pair of matched points and the squared distance between each pair of matched points evaluating topological features on two-dimensional projections of three-dimensional predictions and true values to reduce computational complexity; projecting the predicted minimum intensity defined as the lowest intensity along a given axis, the projection topology loss is computed; The squared distance between each pair of matched points is computed and is expressed as: wherein , , denotes the 1st order Wasserstein distance; the expression of the projection topology loss is: wherein, is the projection topological loss; is the true value is the projection on the given axis; is the predicted value is the projection on the given axis.

2. The multimodal multi-object segmentation method for microvascular decompression according to claim 1, wherein, the taking images of different modalities as inputs of a multi-modal model, sequentially passing through a shared encoder and a specific decoder for feature extraction, and obtaining pixel-level segmentation results and projection features, comprising: taking images of different modalities as inputs of a multi-modal model, and inputting the images of different modalities into a shared encoder after splicing along channels; after the images of different modalities are processed by the shared encoder, the images are split along channels and input into specific decoders corresponding to the modalities of the images respectively, and after feature extraction by the specific decoders, pixel-level segmentation results and projection features are output respectively.

3. The multimodal multi-object segmentation method for microvascular decompression according to claim 1, wherein, the images of different modalities include T2 and TOF modalities; the specific decoders include a T2 specific decoder corresponding to the T2 modality and a TOF specific decoder corresponding to the TOF modality.

4. The microvascular decompression-oriented multi-modal multi-object segmentation method according to claim 1, wherein the expression of the loss function of the multi-modal model is: where, is the dice loss and cross-entropy loss equally weighted sum, is the cross-modal mutual information distillation loss, is the class contrastive loss, is the projection topology loss, is a hyper-parameter to adjust the loss value.

5. A multi-modal multi-object segmentation system for microvascular decompression, the system being configured to implement the multi-modal multi-object segmentation method for microvascular decompression according to any one of claims 1-4, characterized in that, comprising: a feature extraction module, a knowledge distillation module, a class contrast module, a topology preservation constraint module, and a data processing module connected in sequence; wherein the feature extraction module is configured to take images of different modalities as inputs of a multi-modal model, sequentially pass through a shared encoder and a specific decoder for feature extraction, and obtain pixel-level segmentation results and projection features respectively, and store the projection features in a storage library; the knowledge distillation module is configured to calculate a pixel-level segmentation loss function according to the segmentation results, and calculate a cross-modality mutual information distillation loss based on the pixel-level segmentation loss function; The category contrast module includes a contrast learning module with a memory bank, and a category contrast loss is calculated by the contrast learning module according to the projection feature; The topology preservation constraint module is configured to calculate a projection topology loss according to the projection feature; The data processing module is configured to obtain a loss function of the multi-modal model according to the cross-modal mutual information distillation loss, the projection topology loss and the category contrast loss, and perform multi-object segmentation based on the loss function of the multi-modal model.

Images