A lightweight-based multi-modal medical image segmentation method and system

Abstract

The present invention relates to the technical field of medical image segmentation, and particularly to a lightweight multi-modal medical image segmentation method. The steps are as follows: collect medical images to construct a dataset, preprocess the images in the dataset, and divide the preprocessed dataset into a training set and a test set; construct a lightweight medical image segmentation model, the model includes an image encoder, a prompt encoder, and a mask decoder, input the medical images in the training set into the lightweight medical image segmentation model for medical image segmentation to obtain the segmentation result predicted by the model; improve and optimize the lightweight medical image segmentation model to obtain a lightweight medical image segmentation model with updated parameters; input the medical images in the test set into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result. This method can help doctors quickly identify and locate lesions in a clinical environment and improve work efficiency by optimizing computational efficiency and resource utilization.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and particularly to a lightweight multi-modal medical image segmentation method and system. Background Art

[0002] Medical image segmentation is crucial in clinical practice. It can achieve precision medicine by delineating organ boundaries and pathological regions, evaluate treatment effects and disease diagnoses, enhance anatomical understanding and anomaly detection, and help doctors make accurate diagnostic and treatment decisions. With the rapid development of modern medical imaging technology, the quantity and variety of medical images are continuously increasing. These medical images provide rich information for clinical diagnosis, treatment, and research, but at the same time, they also bring challenges in data processing and analysis. The basic task of medical image segmentation is to divide an image into several meaningful regions to better understand the image content. In addition, segmentation technology also plays an important role in surgical planning, radiotherapy, and disease monitoring. However, traditional medical image segmentation methods mostly rely on manually designed features and rules, and usually face problems such as high dependence on expert experience, poor adaptability, and low processing efficiency. With the rise of deep learning technology, especially the widespread application of convolutional neural networks (CNNs), the performance of medical image segmentation has been significantly improved. Deep learning models can automatically learn the features of input data, reduce the dependence on expert knowledge, and have shown excellent effects in various medical image segmentation tasks. Although deep learning has made many progress in medical image segmentation, existing models are often designed for specific tasks or specific types of medical images. When these models are applied to different imaging modalities or new tasks, their performance often drops significantly. Many existing medical image segmentation methods are designed for specific tasks, and when applied to new tasks or different types of imaging data, the performance usually decreases significantly and it is difficult to be generalized to unknown fields. This lack of generality constitutes a substantial obstacle to the widespread application of these methods in clinical practice. Therefore, it is particularly important to develop a segmentation software that can integrate multiple data sources.

[0003] Therefore, the present invention provides a lightweight multi-modal medical image segmentation method and system to solve the above problems. Summary of the Invention

[0004] In view of the deficiencies of the prior art, the present invention has developed a lightweight multi-modal medical image segmentation method and system. The main purpose is to help doctors quickly identify and locate lesions in a clinical environment by optimizing computational efficiency and resource utilization, improve work efficiency, and ultimately improve the treatment and prognosis of patients.

[0005] The technical solution for the present invention to solve the technical problem is a lightweight multi-modal medical image segmentation method and system, which is specifically as follows:

[0006] S1. Collect medical images of multiple imaging modes to construct a dataset, preprocess the images in the dataset, and then divide the preprocessed dataset into a training set and a test set;

[0007] S2. Construct a lightweight medical image segmentation model. The model includes an image encoder, a prompt encoder, and a mask decoder. Input the medical images in the training set into the lightweight medical image segmentation model for medical image segmentation to obtain the segmentation results predicted by the model;

[0008] S3. Improve and optimize the lightweight medical image segmentation model to obtain the lightweight medical image segmentation model with updated parameters;

[0009] S4. Input the medical images in the test set into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation results.

[0010] S1 is specifically as follows:

[0011] S1.1. Construct a dataset:

[0012] The medical images collected in the dataset are all from the CVPR challenge. The data modalities in the CVPR challenge include CT, MRI, X-Ray, Endoscopy, US, and Fundus images, and experts manually perform accurate label annotation on the tissues, organs, and cancer types in the images;

[0013] S1.2. Data preprocessing:

[0014] Convert 3D medical image data into a series of 2D images;

[0015] Perform data augmentation on data of all modalities. The data augmentation operation is to perform horizontal and vertical flipping with a probability of 50%;

[0016] Uniformly adjust and pad the images and labels to a size of 256x256;

[0017] Normalize the images and adjust the intensity values to the range of (0,1);

[0018] S1.3. Divide the dataset:

[0019] Divide the preprocessed dataset into a training set and a test set , , represents the number of data modality types, represents the Sub-datasets of the data modality, and each sub-dataset of each modality contains different numbers of image data corresponding to the modality.

[0020] S2 is as follows:

[0021] S2.1. Construct a lightweight medical image segmentation model, which includes an image encoder, a prompt encoder, and a mask decoder;

[0022] (1) Image encoder: The image encoder sequentially includes 12 lightweight vision Transformer modules LViT, and the 12 LViT are divided into 4 stages. The first stage includes 2 LViT, the second stage includes 2 LViT, the third stage includes 6 LViT, and the fourth stage includes 2 LViT. Each LviT sequentially includes layer normalization Layer Norm, multi-head self-attention Muliti-head Self-Attention, layer normalization Layer Norm, and multi-layer perceptron MLP;

[0023] (2) Prompt encoder: The prompt encoder includes a position embedding layer and a mask downsampling module. The position embedding layer calls to calculate the embedding of the bounding box. The mask downsampling module sequentially includes a convolutional layer Conv, layer normalization Layer Norm, activation function act, convolutional layer Conv, layer normalization Layer Norm, activation function act, and convolutional layer Conv;

[0024] (3) Mask decoder: The mask decoder sequentially includes a feature fusion module, a Transformer module, a Transformer module, a transposed convolution Transposed Conv, a transposed convolution Transposed Conv, and a multi-layer perceptron MLP;

[0025] S2.2. Input the image data in the training set into the lightweight medical image segmentation model. represents the sub-dataset of the training set Any one of the image data in, the image data contains the image and the mask and the mask . The image data is processed by the image encoder to obtain the image feature embedding . The teacher model and feature knowledge distillation technology are introduced to align the image feature embedding to obtain the image feature embedding . The image data is processed by the prompt decoder to obtain the sparse embedding and the dense embedding Embed the image features into , sparse embedding and dense embedding and input them into the mask decoder. After being processed by the mask decoder, the predicted segmentation result is obtained.

[0026] The image segmentation process of the lightweight medical image segmentation model is as follows:

[0027] S2.2.1. Input the image data in the training set into the image encoder. The image encoder reads the image in the image data . The image passes through the LViT in 4 stages in sequence, and finally outputs the image feature embedding ;

[0028] Suppose the data input to the LViT is . The processing process of each LViT for the input data is First, it passes through the first Layer Norm, then the output of the first Layer Norm is input into the Muliti-head Self-Attention, and then the output of the Muliti-head Self-Attention is fused with to obtain . Then, is input into the second LayerNorm, and then the output of the second Layer Norm is input into the MLP. The output of the MLP is fused with to obtain the final output of the LViT ;

[0029] ,

[0030] ,

[0031] Introduce the artificial intelligence model MedSAM dedicated to medical image segmentation. MedSAM includes 12 Transformer modules in sequence. The 12 Transformer modules are divided into four stages, and each stage includes 2, 2, 6, and 2 Transformer modules in sequence. Input the image data in the training set into MedSAM to obtain the image feature embedding ;

[0032] Take MedSAM as the teacher model and the image encoder as the student model, and then introduce the feature knowledge distillation technology to the image feature embedding According to the image feature embedding Perform alignment to obtain the aligned image feature embedding ;

[0033] During the feature knowledge distillation process, the teacher model is supervised and optimized through the combination of binary cross-entropy loss and dice loss, and at the same time The loss function aligns the image feature embeddings of the teacher model and the student model. The objective function of knowledge distillation training is specifically as follows:

[0034] ;

[0035] S2.2.2. Input the image data in the training set into the prompt encoder. The prompt encoder reads the mask in the image data . The prompt encoder generates a prompt bounding box according to the read mask . The prompt bounding box is composed of the coordinates of two corner points, which are and , and represent the horizontal and vertical coordinates of the two corner points respectively. Then, input the coordinates of the two corner points of the prompt bounding box into the boundary embedding module embed_boxes, adjust the coordinates to the pixel center, and perform position encoding on the two corner points through the position embedding layer to generate a sparse embedding aligned with the image embedding feature . The calculation formula is as follows:

[0036] ,

[0037] ,

[0038] where represents the operation of calculating the bounding box embedding, represents the boundary embedding, represents the concatenation operation;

[0039] At the same time, the mask passes through the mask downsampling module and is downsampled through multiple convolutional layers to generate a dense embedding aligned with the image embedding feature . The calculation formula is as follows:

[0040] ,

[0041] where represents the activation function operation;

[0042] S2.2.3. Embed the image embedding features of each image data in the training set , sparse embedding and dense embedding into the decoding masker. First, in the feature fusion module, embed the prompt bounding box IOU token and the mask token embedding and splice them to obtain the fused feature . Then, splice the sparse embedding with the fused feature to obtain the input token . Next, splice the image embedding feature and the dense embedding to obtain the input embedding of the Transformer module ;

[0043] Input the input token and the input embedding into the Transformer module to obtain the updated input embedding . Then, input the updated input embedding into two transposed convolutional layers for upsampling to obtain the image feature embedding . Subsequently, input the image feature embedding into the multi-layer perceptron MLP, and segment the image feature embedding according to the mask token embedding. Finally, the MLP outputs the predicted segmentation result .

[0044] S3 is as follows:

[0045] Calculate the pixel-based, region-based, distribution-based, and shape-based losses based on the predicted segmentation result of the model and the actual segmentation result. Introduce the multi-task balance mechanism MBM to dynamically adjust the weights of the pixel-based, region-based, distribution-based, and shape-based losses. Also introduce the adaptive sharpness norm minimization ASNM strategy to seek the flat minimum of the model by optimizing the maximum loss and the gradient norm. Then, improve and optimize the lightweight medical image segmentation model, select the optimal parameters to update the model, and obtain the lightweight medical image segmentation model with updated parameters;

[0046] The pixel-based MSE loss quantifies the prediction accuracy of the lightweight medical image segmentation model by calculating the pixel value of each pixel in the predicted segmentation result corresponding to the pixel value of the true label. The calculation formula is as follows:

[0047] ,

[0048] Among them, represents the pixel-based MSE loss, represents the predicted segmentation result the total number of pixels in the image, represents the index of represents the th predicted pixel value, represents the th ground truth pixel value;

[0049] The region-based DC loss maintains the accuracy of the shape and boundary of the segmentation region by emphasizing the overlapping region between the predicted segmentation result and the ground truth, and the calculation formula is as follows:

[0050] ,

[0051] Among them, represents the region-based DC loss;

[0052] The distribution-based CE loss ensures the segmentation accuracy by accurately classifying individual pixels of, and the calculation formula is as follows:

[0053] ,

[0054] Among them, represents the distribution-based CE loss;

[0055] The shape-based SD loss enhances the shape features of the segmentation target through the difference between the predicted segmentation result and the mask distance transform map, and the calculation formula is as follows:

[0056] ,

[0057] Among them, represents the shape-based SD loss, represents the total number of channels, represents the index of represents the predicted segmentation result and the th pixel of the mask distance transform map in the channel difference;

[0058] The pixel-based, region-based, distribution-based, and shape-based losses are combined into a resource framework to define MBM, and the calculation formula is as follows:

[0059] ,

[0060] Among them, represent pixel-based, region-based, distribution-based, and shape-based losses represent the number of losses , represent pixel-based MSE loss region-based DC loss distribution-based CE loss shape-based SD loss represent the learnable parameters corresponding to each loss

[0061] The Adaptive Sharpness Norm Minimization (ASNM) strategy is as follows

[0062] Determine the training distributions in the image space and the mask space according to the data in the training set , with point as the center to determine the open ball with radius , represent any point in the open ball, and the open ball is denoted as , ||.|| represents the Euclidean norm

[0063] The overall loss of the sample space is , and the empirical loss function of the sample space is , represents the expectation represents the data distribution represents the average loss of all samples in the sample space represents a pixel point represents the th pixel point represents the total number of pixels

[0064] Calculate the first derivative and Hessian matrix of the empirical loss function at the point , and respectively represent the first derivative and Hessian matrix of the empirical loss function at the point ;

[0065] The optimization process of the ASNM strategy consists of two steps, which are as follows

[0066] Step 1: Find flat minima , for each mini-batch of data, adjust the model parameters in the direction of maximizing the loss, and ASNM finds the perturbation in that maximizes the loss, so that the model parameters are adjusted towards the non-steep regions of the loss landscape, and then update the model parameters. The calculation formula is as follows

[0067] ,

[0068] Among them, represents the overall loss, represents the perturbation, represents finding the non-steep parameter configuration in the loss, represents finding the perturbation that maximizes the loss ;

[0069] Empirical loss function at The first-order flatness at the point is defined as follows:

[0070] ,

[0071] Among them, represents at the first-order flatness at the point, represents the radius of the open ball, represents at the first-order derivative at the point, represents taking the maximum point in the open ball;

[0072] Second step: Approximately optimize by calculating the gradient of the first-order flatness at the point to optimize the maximum eigenvalue of the Hessian matrix, making the curvature at the minimum point in the perturbation domain smoother. The calculation formula is as follows:

[0073] ,

[0074] Among them,

[0075] ,

[0076] Among them, represents taking the derivative of the first-order flatness, represents the function the first-order derivative of, represents adjusting the gradient of the loss function.

[0077] The present invention also provides a lightweight multi-modal medical image segmentation system, which executes a lightweight multi-modal medical image segmentation method, including the following modules:

[0078] Data acquisition module: Collect medical images of multiple imaging modes to construct a data set;

[0079] Data preprocessing module: Preprocess the data set in the data acquisition module and divide it into a training set and a test set;

[0080] Lightweight medical image segmentation module: Input the training set in the data domain processing module into the lightweight medical image segmentation module, train the constructed lightweight medical image segmentation model, and obtain the predicted segmentation result;

[0081] Optimization module: Improve and optimize the lightweight medical image segmentation model based on the loss function and ASNM strategy to obtain the lightweight medical image segmentation model with updated parameters. Input the test set in the data preprocessing module into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

[0082] The effects provided in the invention content are only the effects of the embodiments, rather than all the effects of the invention. The above technical solutions have the following advantages or beneficial effects:

[0083] The present invention proposes a lightweight-based multi-modal medical image segmentation method and system to reduce the degree of resource dependence and improve the model segmentation accuracy and generalization performance; LViT in the present invention is a scaled-down vision transformer using a progressive shrinking method. By using LViT to construct an image encoder, the model parameters can be significantly reduced. In addition, the present invention ensures the segmentation accuracy of the lightweight model by introducing feature knowledge distillation, extracts knowledge from the well-trained MedSAM using the teacher-student framework, and transfers the knowledge of the teacher model encoder to the LViT encoder, effectively transmitting the rich features and decision-making information in the teacher model to the student model;

[0084] The present invention also strengthens the attention to high-dimensional features through a multi-task balance mechanism, enhances the model's learning ability for difficult tasks, reduces the uncertainty of the prediction results, and captures more detailed information through high-dimensional features; in order to prevent the model from falling into sharp minima in the loss space, the present invention introduces the adaptive sharpness norm minimization (ASNM) strategy. By optimizing the maximum loss and gradient norm, it seeks the flat minima of the model, enabling the model to have excellent generalization ability while capturing detailed information. The aim is to make the model comprehensively learn the diversity of data and improve the model generalization performance by focusing on the flatness of the loss function and the normalization processing of the gradient.

[0085] In summary, by optimizing the computational efficiency and resource utilization, the present invention can help doctors quickly identify and locate lesions in the clinical environment, improve work efficiency, and ultimately improve the treatment and prognosis of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention.

[0087] Figure 1 This is a schematic diagram of the overall framework of the method of the present invention.

[0088] Figure 2 This is the experimental result diagram of the present invention.

[0089] Figure 3 This is a comparison diagram of the experimental results between the method of the present invention and the existing methods. Specific embodiments

[0090] In order to clearly illustrate the technical features of the present solution, the present invention will be elaborated in detail below through specific embodiments and in conjunction with its accompanying drawings.

[0091] Example 1

[0092] As Figure 1 shown, the present invention proposes a lightweight multi-modal medical image segmentation method, specifically as follows:

[0093] S1. Collect medical images of multiple imaging modalities to construct a dataset, preprocess the images in the dataset, and then divide the preprocessed dataset into a training set and a test set;

[0094] S2. Construct a lightweight medical image segmentation model. The model includes an image encoder, a prompt encoder, and a mask decoder. Input the medical images in the training set into the lightweight medical image segmentation model for medical image segmentation to obtain the segmentation results predicted by the model;

[0095] S3. Improve and optimize the lightweight medical image segmentation model to obtain a lightweight medical image segmentation model with updated parameters;

[0096] S4. Input the medical images in the test set into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation results.

[0097] S1 is specifically as follows:

[0098] S1.1. Construct a dataset:

[0099] The medical images collected in the dataset are all from the CVPR challenge. The data modalities in the CVPR challenge include CT, MRI, X-Ray, Endoscopy, US, and Fundus images, and experts manually perform accurate label annotation on the tissues, organs, and cancer types in the images;

[0100] S1.2. Data preprocessing:

[0101] Convert 3D medical image data into a series of 2D images;

[0102] Perform data augmentation on all modal data. The data augmentation operation is to perform horizontal and vertical flipping with a probability of 50%.

[0103] Uniformly adjust and pad the images and labels to a size of 256x256.

[0104] Normalize the images and adjust the intensity values to the range of (0,1).

[0105] S1.3. Divide the dataset:

[0106] Divide the preprocessed dataset into a training set and a test set , , represents the number of data modal types, represents the sub-dataset of the data modality in, and each sub-dataset of each modality contains different numbers of image data corresponding to the modality.

[0107] S2 is as follows:

[0108] S2.1. Build a lightweight medical image segmentation model, which includes an image encoder, a prompt encoder, and a mask decoder;

[0109] (1) Image encoder: The image encoder sequentially includes 12 lightweight vision Transformer modules LViT. Divide the 12 LViT into 4 stages. The first stage includes 2 LViT, the second stage includes 2 LViT, the third stage includes 6 LViT, and the fourth stage includes 2 LViT. Each LviT sequentially includes layer normalization Layer Norm, multi-head self-attention Muliti-head Self-Attention, layer normalization Layer Norm, and multi-layer perceptron MLP;

[0110] (2) Prompt encoder: The prompt encoder includes a position embedding layer and a mask downsampling module. The position embedding layer calls to calculate the embedding of the bounding box. The mask downsampling module sequentially includes a convolutional layer Conv, layer normalization Layer Norm, activation function act, convolutional layer Conv, layer normalization Layer Norm, activation function act, and convolutional layer Conv;

[0111] (3) Mask decoder: The mask decoder sequentially includes a feature fusion module, a Transformer module, a Transformer module, a transposed convolution Transposed Conv, a transposed convolution Transposed Conv, and a multi-layer perceptron MLP;

[0112] S2.2. Input the image data in the training set into the lightweight medical image segmentation model, denote a sub-dataset of the training set and any one of the image data in it. The image data contains an image and a mask . The image data is processed by the image encoder to obtain an image feature embedding . The teacher model and feature knowledge distillation technology are introduced to align the image feature embedding to obtain an image feature embedding . The image data is processed by the prompt decoder to obtain a sparse embedding and a dense embedding . The image feature embedding , the sparse embedding and the dense embedding are input into the mask decoder, and the predicted segmentation result is obtained after being processed by the mask decoder.

[0113] The image segmentation process of the lightweight medical image segmentation model is specifically as follows:

[0114] S2.2.1. Input the image data in the training set into the image encoder. The image encoder reads the image in the image data . The image successively passes through 4 stages of LViT and finally outputs an image feature embedding ;

[0115] Let the data input into LViT be . The processing process of each LViT for the input data is First, it passes through the first Layer Norm, then the output of the first Layer Norm is input into the Muliti-head Self-Attention, and then the output of the Muliti-head Self-Attention is fused with to obtain . is input into the second LayerNorm, and then the output of the second Layer Norm is input into the MLP. The output of the MLP is fused with to obtain the final output of LViT;

[0116] ,

[0117] ,

[0118] Introduce the artificial intelligence model MedSAM dedicated to medical image segmentation. MedSAM includes 12 Transformer modules in sequence. The 12 Transformer modules are divided into four stages, and each stage includes 2, 2, 6, and 2 Transformer modules in sequence. The image data in the training set is input into MedSAM to obtain image feature embeddings ;

[0119] Take MedSAM as the teacher model and the image encoder as the student model. Then introduce the feature knowledge distillation technology to align the image feature embeddings According to the image feature embeddings to obtain the aligned image feature embeddings ;

[0120] During the feature knowledge distillation process, the teacher model is supervised and optimized through the combination of binary cross-entropy loss and dice loss. At the same time, the loss function is used to align the image feature embeddings of the teacher model and the student model. The objective function of knowledge distillation training is specifically as follows:

[0121] ;

[0122] S2.2.2. Input the image data in the training set into the prompt encoder. The prompt encoder reads the mask in the image data . The prompt encoder generates a prompt bounding box according to the read mask . The prompt bounding box is composed of two corner coordinates. The two corners are respectively and . and represent the horizontal and vertical coordinates of the two corners respectively. Then, the two corner coordinates of the prompt bounding box are input into the boundary embedding module embed_boxes to adjust the coordinates to the pixel center, and the two corners are position-encoded through the position embedding layer to generate a sparse embedding aligned with the image embedding features . The calculation formula is as follows:

[0123] ,

[0124] ,

[0125] Among them, represents the operation of calculating the bounding box embedding, represents the boundary embedding, represents the concatenation operation;

[0126] Meanwhile, the mask passes through the mask downsampling module and is downsampled through multiple convolutional layers to generate a dense embedding aligned with the image embedding feature , and the calculation formula is as follows:

[0127] ,

[0128] Among them, represents the activation function operation;

[0129] S2.2.3. Input the image embedding feature , sparse embedding and dense embedding of each image data in the training set into the decoding masker. First, in the feature fusion module, the prompt bounding box IOU token embedding and the mask token embedding are concatenated to obtain the fusion feature , then the sparse embedding is concatenated with the fusion feature to obtain the input token , and then the image embedding feature and the dense embedding are concatenated to obtain the input embedding of the Transformer module;

[0130] Input the input token and the input embedding into the Transformer module to obtain the updated input embedding , and then the updated input embedding is input into two transposed convolutional layers for upsampling to obtain the image feature embedding , and subsequently, the image feature embedding is input into the multi-layer perceptron MLP. According to the mask token embedding , the image feature embedding is segmented, and finally, the MLP outputs the predicted segmentation result .

[0131] S3 is specifically as follows:

[0132] Calculate the pixel-based, region-based, distribution-based, and shape-based losses based on the predicted segmentation results and the actual segmentation results of the model. Introduce the multi-task balance mechanism MBM to dynamically adjust the weights of the pixel-based, region-based, distribution-based, and shape-based losses. Also introduce the adaptive sharpness norm minimization ASNM strategy to seek the flat minimum of the model by optimizing the maximum loss and the gradient norm, thereby improving and optimizing the lightweight medical image segmentation model. Select the optimal parameters to update the model and obtain the lightweight medical image segmentation model with updated parameters;

[0133] The pixel-based MSE loss quantifies the prediction accuracy of the lightweight medical image segmentation model by calculating the pixel value of each pixel in the predicted segmentation result and the corresponding pixel value of the true label. The calculation formula is as follows:

[0134] ,

[0135] where represents the pixel-based MSE loss, represents the predicted segmentation result the total number of pixels in the image, represents the index of represents the th predicted pixel value, represents the th true label pixel value;

[0136] The region-based DC loss maintains the accuracy of the shape and boundary of the segmentation region by emphasizing the overlapping region between the predicted segmentation result and the true label. The calculation formula is as follows:

[0137] ,

[0138] where represents the region-based DC loss;

[0139] The distribution-based CE loss ensures the segmentation accuracy by accurately classifying individual pixels of. The calculation formula is as follows:

[0140] ,

[0141] where represents the distribution-based CE loss;

[0142] The shape-based SD loss enhances the shape features of the segmentation target by the difference between the predicted segmentation result and the mask distance transform map. The calculation formula is as follows:

[0143] ,

[0144] Among them, represents the shape-based SD loss, represents the total number of channels, represents the index of represents the predicted segmentation result and the difference between the th pixel of the mask distance transform map in the channel ;

[0145] Combine the pixel-based, region-based, distribution-based, and shape-based losses into a resource framework to define MBM. The calculation formula is as follows:

[0146] ,

[0147] Among them, represents the pixel-based, region-based, distribution-based, and shape-based losses, represents the number of losses, , represents the pixel-based MSE loss, the region-based DC loss, the distribution-based CE loss, the shape-based SD loss, represents the learnable parameter corresponding to each loss.

[0148] The adaptive sharpness norm minimization ASNM strategy is as follows:

[0149] Determine the training distributions of the image space and the mask space according to the data in the training set , with the point as the center and as the radius to determine the open ball , represents any point in the open ball, and the open ball is represented as , ||.|| represents the Euclidean norm;

[0150] The overall loss of the sample space is , and the empirical loss function of the sample space is , represents the expectation, represents the data distribution, represents the average loss of all samples in the sample space, represents the pixel point, represents the th pixel point, represents the total number of pixels;

[0151] Calculate the empirical loss function At the first-order derivative and Hessian matrix of the point, and respectively represent the first-order derivative and Hessian matrix of the empirical loss function at the point;

[0152] The optimization process of the ASNM strategy includes two steps, which are as follows:

[0153] The first step: Find the flat minimum , for each mini-batch of data, adjust the model parameters to the direction of maximum loss, and ASNM finds the perturbation that maximizes the loss in , so that the model parameters are adjusted towards the non-steep region of the loss landscape, and then update the model parameters. The calculation formula is as follows:

[0154] ,

[0155] where, represents the overall loss, represents the perturbation, represents finding the non-steep parameter configuration in the loss, represents finding the perturbation that maximizes the loss ;

[0156] The empirical loss function at the point is defined as the first-order flatness as follows:

[0157] ,

[0158] where, represents at the first-order flatness at the point, represents the radius of the open ball, represents at the first-order derivative of the point, represents taking the maximum point in the open ball;

[0159] The second step: Approximately optimize by calculating the gradient of the first-order flatness at the point to optimize the maximum eigenvalue of the Hessian matrix, so that the curvature at the minimum point in the perturbation domain is smoother. The calculation formula is as follows:

[0160] ,

[0161] where,

[0162] ,

[0163] Among them, represents the derivative of the first-order flatness, represents the function of the first-order derivative, represents the gradient adjustment of the loss function.

[0164] Example 2

[0165] The present invention also provides a lightweight multi-modal medical image segmentation system that executes a lightweight multi-modal medical image segmentation method, including the following modules:

[0166] Data acquisition module: Collect medical images of multiple imaging modes to construct a data set;

[0167] Data preprocessing module: Preprocess the data set in the data acquisition module and divide it into a training set and a test set;

[0168] Lightweight medical image segmentation module: Input the training set in the data domain processing module into the lightweight medical image segmentation module, train the constructed lightweight medical image segmentation model, and obtain the predicted segmentation result;

[0169] Optimization module: Improve and optimize the lightweight medical image segmentation model based on the loss function and the ASNM strategy, obtain the lightweight medical image segmentation model with updated parameters, input the test set in the data preprocessing module into the lightweight medical image segmentation model with updated parameters, and obtain the final predicted segmentation result.

[0170] Example 3

[0171] As Figure 2 shown, the three groups of pictures are abdominal CT, intestinal polyps, and chest X-Ray images respectively. In each group, the images from left to right are the input, label, and predicted result. The input images in the three groups of images are processed by the method in the present invention. The first group of abdominal CT predicts the liver, spleen, pancreas, left and right kidneys. The second group predicts intestinal polyps. The third group predicts the left and right lungs.

[0172] By Figure 2From the prediction results, it can be seen that the network model in the present invention shows satisfactory performance in cross-modal object automatic segmentation, demonstrating strong stability and consistency, with a low mis-segmentation rate. The segmentation accuracy of the network model in the present invention is very high, capable of accurately identifying and segmenting the regions of interest. For the precise segmentation of the target task, the model also exhibits good general performance and can effectively perform cross-modal segmentation of the regions of interest. This indicates that although the cross-modal segmentation of medical images poses greater challenges, the network model in the present invention can still provide reliable prediction results and shows high stability and accuracy.

[0173] Example 4

[0174] To prove that the present invention is superior to the existing methods, the input images in three groups of pictures in Figure 2 are respectively processed by the method of the present invention and the existing methods (U-Mamba represents a general network for biomedical image segmentation, nnUNet represents a deep learning model based on the U-Net architecture, nnFormer represents a 3D medical image segmentation network based on the cross-Transformer structure, SwinUNETR represents a combination of window Transformer and U-Net, and U-MedSAM represents an arbitrary segmentation network for medical images based on uncertainty), and the segmentation results as shown in Figure 3 are obtained. From the comparison of the experimental results in Figure 3 , it can be seen that the method of the present invention performs excellently in detail processing and can completely and accurately segment the target organ. Even in the second group of comparison images with poor contrast, the method of the present invention can still maintain precise segmentation ability, while other models perform poorly in capturing complex boundary features. As shown by the yellow arrow in Figure 3 , this will lead to the prediction result not matching the lesion area, showing an obvious phenomenon of insufficient segmentation;

[0175] At the same time, the method of the present invention is compared with the existing methods in terms of the Dice similarity coefficient and the surface normalized distance. The Dice similarity coefficient is used to evaluate the degree of regional overlap between the mask marked by experts and the segmentation result, and the larger the value, the higher the degree of overlap; the surface normalized distance mainly evaluates the boundary consistency between the mask marked by experts and the segmentation result. As shown in Table 1, compared with other existing methods, the method of the present invention can be optimized and improved to adapt to different image features, accurately capture the subtle structural changes in medical images, and thus significantly improve the segmentation accuracy;

[0176] Table 1 Comparison results between the method of the present invention and the existing methods

[0177]

[0178] In summary, it can be proven that the lightweight multi-modal medical image segmentation framework proposed by the present invention has significant advantages over the prior art. First, through the bounding box hint and knowledge distillation techniques, the model parameters are greatly reduced, thereby reducing the computational complexity, which enables it to operate efficiently in resource-constrained environments. In addition, the framework has flexible adaptability. By adopting a multi-task balancing mechanism to dynamically adjust the model learning resources, it can effectively respond to demand changes according to the priorities of different tasks, further optimizing the segmentation effect. Introducing the ASNM training optimization strategy also speeds up the model convergence rate, enabling it to quickly adapt to different types of medical images, thus improving the training efficiency. Finally, the lightweight and efficient design makes the deployment of this model more convenient in practical applications and suitable for rapid application in clinical environments. In summary, these advantages make the method in the present invention show important application potential and practical value in the field of medical image segmentation.

[0179] Although the specific implementation manners of the invention are described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of the present invention. Based on the technical solutions of the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts are still within the protection scope of the present invention.

Claims

1. A lightweight multimodal medical image segmentation method, characterized in that: The following steps are involved: S1. Collect medical images of various imaging modes to build a data set, preprocess the images in the data set, and then divide the preprocessed data set into a training set and a test set; S2. Construct a lightweight medical image segmentation model, which includes an image encoder, a prompt encoder, and a mask decoder. Input the medical images in the training set into the lightweight medical image segmentation model to perform medical image segmentation and obtain the segmentation results predicted by the model. S3, improving and optimizing the lightweight medical image segmentation model to obtain a lightweight medical image segmentation model with updated parameters; S4, inputting the medical images in the test set into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result; S3 is as follows: According to the segmentation results predicted by the model and the actual segmentation results, pixel-based, region-based, distribution-based and shape-based losses are calculated, and a multi-task balancing mechanism MBM is introduced to dynamically adjust the weights of pixel-based, region-based, distribution-based and shape-based losses. An adaptive sharpness norm minimization ASNM strategy is also introduced to seek the flat minimum of the model by optimizing the maximum loss and gradient norm, thereby improving and optimizing the lightweight medical image segmentation model, selecting the optimal parameters to update the model, and obtaining a lightweight medical image segmentation model after parameter update; The pixel-based MSE loss is calculated by calculating the predicted segmentation result. The prediction accuracy of the lightweight medical image segmentation model is quantified by comparing each pixel value in the image with the pixel value corresponding to the true label. The calculation formula is as follows: ， in, represents the pixel-based MSE loss, Represents the predicted segmentation result The total number of pixels in the image, express The index of Indicates The predicted pixel values, Indicates The pixel value of the true label; The region-based DC loss emphasizes the predicted segmentation results. The overlapping area between the real label and the real label is used to maintain the accuracy of the shape and boundary of the segmented area. The calculation formula is as follows: ， in, represents the DC loss based on the area; The distribution-based CE loss ensures segmentation accuracy by accurately classifying individual pixels. The calculation formula is as follows: ， in, represents the distribution-based CE loss; Shape-based SD loss passes the predicted segmentation result The difference between the mask distance transform map and the image is used to enhance the shape features of the segmented target. The calculation formula is as follows: ， in, represents the shape-based SD loss, Indicates the total number of channels, express The index of Represents the predicted segmentation result The distance transformation map with the mask Pixels between channels The difference between The pixel-based, region-based, distribution-based, and shape-based losses are combined into a resource framework to define MBM, which is calculated as follows: ， in, represents pixel-based, region-based, distribution-based, and shape-based losses, represents the number of losses, , represents the pixel-based MSE loss, Based on the DC loss of the area, Distribution-based CE loss, Shape-based SD loss, Represents the learnable parameters corresponding to each loss.

2. The lightweight multimodal medical image segmentation method according to claim 1, characterized in that: S1 is as follows: S1.

1. Constructing the dataset: The medical images collected in the dataset are all from the CVPR challenge. The data modalities in the CVPR challenge include CT, MRI, X-Ray, Endoscopy, US, and Fundus images. Experts manually annotate the tissues, organs, and cancer types in the images with precise labels. S1.2, Data preprocessing: Convert 3D medical imaging data into a series of 2D images; Data augmentation is performed on all modal data, and the data augmentation operation is horizontal and vertical flipping with a probability of 50%; Resize and pad the image and label to 256x256 size. Normalize the image and adjust the intensity value to the range of (0,1); S1.

3. Divide the data set: The preprocessed data set is divided into training sets in a ratio of 4:

1. and test set , , Indicates the number of data modal types, Indicates The sub-datasets of the data modalities in the data set are each sub-dataset of a modality, and each sub-dataset of a modality contains a different number of image data of the corresponding modality.

3. The lightweight multimodal medical image segmentation method according to claim 2, characterized in that: S2 is as follows: S2.

1. Construct a lightweight medical image segmentation model, which includes an image encoder, a hint encoder, and a mask decoder; (1) Image encoder: The image encoder includes 12 lightweight visual Transformer modules LViT in sequence. The 12 LViTs are divided into 4 stages. The first stage includes 2 LViTs, the second stage includes 2 LViTs, the third stage includes 6 LViTs, and the fourth stage includes 2 LViTs. Each LViT includes layer normalization Layer Norm, multi-head self-attention, layer normalization Layer Norm and multi-layer perceptron MLP in sequence. (2) Hint encoder: The hint encoder consists of a position embedding layer and a mask downsampling module. The position embedding layer calls To calculate the embedding of the bounding box, the mask downsampling module includes the convolution layer Conv, the layer normalization Layer Norm, the activation function act, the convolution layer Conv, the layer normalization Layer Norm, the activation function act and the convolution layer Conv in sequence; (3) Mask decoder: The mask decoder includes feature fusion module, Transformer module, Transformer module, deconvolution Transposed Conv, deconvolution Transposed Conv and multi-layer perceptron MLP in sequence; S2.

2. Image data in the training set Input to the lightweight medical image segmentation model, Represents the training set Sub-dataset of Any image data in Include images and mask , image data Image feature embedding is obtained after processing by the image encoder , introduce teacher model and feature knowledge distillation technology to embed image features Align and get image feature embedding , image data After processing by the hint decoder, the sparse embedding is obtained and dense embedding , embedding image features into , sparse embedding and dense embedding Input to the mask decoder, and the predicted segmentation result is obtained after processing by the mask decoder.

4. The lightweight multimodal medical image segmentation method according to claim 3, characterized in that: The image segmentation process of the lightweight medical image segmentation model is as follows: S2.2.

1. Image data in the training set Input to the image encoder, the image encoder reads the image data Images in ,image After four stages of LViT, the final output image feature embedding ; Assume that the input data of LViT is , the processing process of each LViT on the input data is First, it passes through the first Layer Norm, then the first Layer Norm output is input into Multi-head Self-Attention, and then the output of Multi-head Self-Attention is combined with Fusion, get ,Will Input to the second LayerNorm, and then input the output of the second LayerNorm into the MLP, and compare the output of the MLP with Fusion is performed to obtain the final output of LViT ; ， ， MedSAM, an artificial intelligence model specifically for medical image segmentation, is introduced. MedSAM includes 12 Transformer modules in sequence. The 12 Transformer modules are divided into four stages. Each stage includes 2, 2, 6, and 2 Transformer modules in sequence. Input into MedSAM to get image feature embedding ; MedSAM is used as the teacher model, the image encoder is used as the student model, and then the feature knowledge distillation technology is introduced to embed the image features Embedding based on image features Align and get the aligned image feature embedding ; During the feature knowledge distillation process, the teacher model is supervised and optimized through a combination of binary cross-entropy loss and dice loss. The loss function aligns the image feature embeddings of the teacher model and the student model, and the objective function of knowledge distillation training The details are as follows: ； S2.2.

2. The image data in the training set Input to prompt encoder, prompt encoder to read image data The mask in , prompting the encoder to read the mask Generate a hint bounding box, which is composed of two corner point coordinates. and , and Represent the horizontal and vertical coordinates of the two corner points respectively, and then input the coordinates of the two corner points of the prompt bounding box into the boundary embedding module embed_boxes, adjust the coordinates to the pixel center, and encode the positions of the two corner points through the position embedding layer to generate the same image embedding features. Aligned Sparse Embedding , the calculation formula is as follows: ， ， in, represents the operation of computing bounding box embeddings, represents boundary embedding, Represents a splicing operation; At the same time, the mask Through the mask downsampling module, after multiple convolutional layers of downsampling, the generated image embedding features are generated. Aligned dense embedding , the calculation formula is as follows: ， in, Represents the activation function operation; S2.2.

3. Embedding features into images for each image in the training set , sparse embedding and dense embedding Input to the decoder masker, first embed the hint bounding box IOU token in the feature fusion module and mask token embedding Splice to get fusion features , and then sparse embedding Re- and fusion features Concatenate to get the input token , and then embed the image into features and dense embedding Concatenate to get the input embedding of the Transformer module ; Enter the token and input embedding Input Transformer module to get updated input embedding , and then embed the updated input Input to two deconvolution Transposed Conv for upsampling to get image feature embedding , and then the image features are embedded Input into the multi-layer perceptron MLP, according to the mask token embedding Embedding image features Perform segmentation. Finally, MLP outputs the predicted segmentation results. .

5. The lightweight multimodal medical image segmentation method according to claim 4, characterized in that: The specific ASNM strategy for minimizing the sharpness norm is as follows: Determine the training distribution of image space and mask space based on the data in the training set , with a point Center Determine the open sphere for the radius , represents any point in the open sphere, and the open sphere is represented by ,||.|| represents the Euclidean norm; The overall loss of the sample space is , the empirical loss function of the sample space is , Express expectations, represents the data distribution, represents the average loss of all samples in the sample space, Represents a pixel, Indicates pixels, Indicates the total number of pixels; Calculate the empirical loss function exist The first-order derivative and Hessian matrix of the point, and Represent the empirical loss function respectively exist First-order derivatives and Hessian matrices of points; The optimization process of the ASNM strategy includes two steps, as follows: Step 1: Finding Flat Minima For each small batch of data, the model parameters are adjusted to maximize the loss. ASNM finds The perturbation that maximizes the loss , so that the model parameters are adjusted towards the non-steep area of ​​the loss landscape, and then the model parameters are updated. The calculation formula is as follows: ， in, represents the overall loss, represents disturbance, represents the search for a non-steep parameter configuration in the loss, It means finding the perturbation that maximizes the loss ; Experience loss function exist The first-order flatness at a point is defined as follows: ， in, express exist First-order flatness at the point, represents the radius of the open sphere, Indicated in The first derivative of a point, It means taking the maximum point in the open ball; Step 2: Approximate The optimization is performed by calculating the first-order flatness at the point The gradient at the point optimizes the maximum eigenvalue of the Hessian matrix, making the curvature at the minimum point in the perturbation field smoother. The calculation formula is as follows: ， in, ， in, represents the derivative of first-order flatness, Representation function The first derivative of Indicates gradient adjustment of the loss function.

6. A lightweight multimodal medical image segmentation system, executing a lightweight multimodal medical image segmentation method according to any one of claims 1 to 5, characterized in that: Includes the following modules: Data acquisition module: collects medical images of various imaging modes to build data sets; Data preprocessing module: preprocess the data set in the data acquisition module and divide it into training set and test set; Lightweight medical image segmentation module: input the training set in the data domain processing module into the lightweight medical image segmentation module, train the constructed lightweight medical image segmentation model, and obtain the predicted segmentation result; Optimization module: Based on the loss function and ASNM strategy, the lightweight medical image segmentation model is improved and optimized to obtain the lightweight medical image segmentation model with updated parameters. The test set in the data preprocessing module is input into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

Images