Lightweight-based multi-modal medical image segmentation method and system

Abstract

The invention relates to the technical field of medical image segmentation, in particular to a multi-modal medical image segmentation method based on lightweight. The method comprises the following steps: collecting medical images to construct a data set, preprocessing the images in the data set, and dividing the preprocessed data set into a training set and a test set; a lightweight medical image segmentation model is constructed, the model comprises an image encoder, a prompt encoder and a mask decoder, the medical images in the training set are input into the lightweight medical image segmentation model for medical image segmentation, and a segmentation result predicted by the model is obtained; the lightweight medical image segmentation model is improved and optimized, and the lightweight medical image segmentation model after parameter updating is obtained; and inputting the medical images in the test set into the parameter-updated lightweight medical image segmentation model to obtain a final predicted segmentation result. According to the method, by optimizing the calculation efficiency and resource utilization, doctors can be helped to quickly identify and position lesions in a clinical environment, and the working efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a lightweight multimodal medical image segmentation method and system. Background Art

[0002] Medical image segmentation is crucial in clinical practice. It can achieve precision medicine by depicting organ boundaries and pathological areas, evaluate treatment effects and disease diagnosis, enhance anatomical understanding and abnormality detection, and help doctors make accurate diagnosis and treatment decisions. With the rapid development of modern medical imaging technology, the number and types of medical images are increasing. These medical images provide rich information for clinical diagnosis, treatment and research, but also bring challenges in data processing and analysis. The basic task of medical image segmentation is to divide the image into several meaningful areas in order to better understand the image content. In addition, segmentation technology also plays an important role in surgical planning, radiotherapy and disease monitoring. However, traditional medical image segmentation methods mostly rely on manually designed features and rules, and usually face problems such as high dependence on expert experience, poor adaptability and low processing efficiency. With the rise of deep learning technology, Since the widespread application of deep learning, especially convolutional neural networks (CNN), the performance of medical image segmentation has been significantly improved. Deep learning models can automatically learn the characteristics of input data, reduce dependence on expert knowledge, and show superior results in various medical image segmentation tasks. Although deep learning has made many advances in medical image segmentation, existing models are often designed for specific tasks or specific types of medical images. When these models are applied to different imaging modalities or new tasks, their performance often decreases significantly. Many existing medical image segmentation methods are designed for specific tasks. When applied to new tasks or different types of imaging data, the performance is usually significantly reduced and difficult to generalize to unknown areas. This lack of versatility poses a substantial obstacle to the widespread application of these methods in clinical practice. Therefore, it is particularly important to develop a segmentation software that can integrate multiple data sources. Therefore, the present invention provides a lightweight multimodal medical image segmentation method and system to solve the above problems. Summary of the invention

[0003] In view of the shortcomings of the existing technology, the present invention has developed a lightweight multimodal medical image segmentation method and system. The main purpose is to help doctors quickly identify and locate lesions in a clinical environment by optimizing computing efficiency and resource utilization, thereby improving work efficiency and ultimately improving the treatment and prognosis of patients.

[0004] The technical solution to the technical problem of the present invention is a lightweight multimodal medical image segmentation method and system, which are as follows: S1. Collect medical images of various imaging modes to build a data set, preprocess the images in the data set, and then divide the preprocessed data set into a training set and a test set; S2. Construct a lightweight medical image segmentation model, which includes an image encoder, a hint encoder, and a mask decoder. Input the medical images in the training set into the lightweight medical image segmentation model to perform medical image segmentation and obtain the segmentation results predicted by the model. S3, improving and optimizing the lightweight medical image segmentation model to obtain a lightweight medical image segmentation model with updated parameters; S4. Input the medical images in the test set into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

[0005] S1 is as follows: S1.1. Constructing the dataset: The medical images collected in the dataset are all from the CVPR challenge. The data modalities in the CVPR challenge include CT, MRI, X-Ray, Endoscopy, US, and Fundus images. Experts manually label the tissues, organs, and cancer types in the images accurately. S1.2, Data preprocessing: Convert 3D medical imaging data into a series of 2D images; Data augmentation is performed on all modal data, and the data augmentation operation is horizontal and vertical flipping with a probability of 50%; Resize and pad the image and label to 256x256 size. Normalize the image and adjust the intensity value to the range of (0,1); S1.3. Divide the data set: The preprocessed data set is divided into training sets in a ratio of 4:1. and test set , , Indicates the number of data modal types, Indicates The sub-datasets of the data modalities in the data set are each sub-dataset of a modality, and each sub-dataset of a modality contains a different number of image data of the corresponding modality.

[0006] S2 is as follows: S2.1. Construct a lightweight medical image segmentation model, which includes an image encoder, a hint encoder, and a mask decoder; (1) Image encoder: The image encoder includes 12 lightweight visual Transformer modules LViT in sequence. The 12 LViTs are divided into 4 stages. The first stage includes 2 LViTs, the second stage includes 2 LViTs, the third stage includes 6 LViTs, and the fourth stage includes 2 LViTs. Each LViT includes layer normalization Layer Norm, multi-head self-attention, layer normalization Layer Norm and multi-layer perceptron MLP in sequence. (2) Hint encoder: The hint encoder consists of a position embedding layer and a mask downsampling module. The position embedding layer calls To calculate the embedding of the bounding box, the mask downsampling module includes the convolution layer Conv, the layer normalization Layer Norm, the activation function act, the convolution layer Conv, the layer normalization Layer Norm, the activation function act and the convolution layer Conv in sequence; (3) Mask decoder: The mask decoder includes feature fusion module, Transformer module, Transformer module, deconvolution Transposed Conv, deconvolution Transposed Conv and multi-layer perceptron MLP in sequence; S2.2. Image data in the training set Input to the lightweight medical image segmentation model, Represents the training set Sub-dataset Any image data in Include images and mask , image data Image feature embedding is obtained after processing by the image encoder , introduce teacher model and feature knowledge distillation technology to embed image features Align and get image feature embedding , image data After processing by the hint decoder, the sparse embedding is obtained and dense embedding , embedding image features into , sparse embedding and dense embedding Input to the mask decoder, and the predicted segmentation result is obtained after processing by the mask decoder.

[0007] The image segmentation process of the lightweight medical image segmentation model is as follows: S2.2.1. Image data in the training set Input to the image encoder, the image encoder reads the image data Images in ,image After four stages of LViT, the final output image feature embedding ; Assume that the input data of LViT is , the processing process of each LViT on the input data is First, it passes through the first Layer Norm, then the first Layer Norm output is input into Multi-head Self-Attention, and then the output of Multi-head Self-Attention is combined with Fusion, get ,Will Input to the second LayerNorm, and then input the output of the second LayerNorm into the MLP, and compare the output of the MLP with Fusion is performed to obtain the final output of LViT ; , , MedSAM, an artificial intelligence model specifically for medical image segmentation, is introduced. MedSAM includes 12 Transformer modules in sequence. The 12 Transformer modules are divided into four stages. Each stage includes 2, 2, 6, and 2 Transformer modules in sequence. Input into MedSAM to get image feature embedding ; MedSAM is used as the teacher model, the image encoder is used as the student model, and then the feature knowledge distillation technology is introduced to embed the image features Embedding based on image features Align and get the aligned image feature embedding ; During the feature knowledge distillation process, the teacher model is supervised and optimized through a combination of binary cross-entropy loss and dice loss. The loss function aligns the image feature embeddings of the teacher model and the student model, and the objective function of knowledge distillation training The details are as follows: ; S2.2.2. The image data in the training set Input to prompt encoder, prompt encoder to read image data The mask in , prompting the encoder to read the mask Generate a hint bounding box, which is composed of two corner point coordinates. and , and Represent the horizontal and vertical coordinates of the two corner points respectively, and then input the coordinates of the two corner points of the prompt bounding box into the boundary embedding module embed_boxes, adjust the coordinates to the pixel center, and encode the positions of the two corner points through the position embedding layer to generate the same image embedding features. Aligned Sparse Embedding , the calculation formula is as follows: , , in, represents the operation of computing bounding box embeddings, represents boundary embedding, Represents a splicing operation; At the same time, the mask Through the mask downsampling module, after multiple convolutional layers of downsampling, the generated image embedding features are generated. Aligned Dense Embedding , the calculation formula is as follows: , in, Represents the activation function operation; S2.2.3. Embedding features into images for each image in the training set , sparse embedding and dense embedding Input to the decoder masker, first embed the hint bounding box IOU token in the feature fusion module and mask token embedding Splice to get fusion features , and then sparse embedding Re- and fusion features Concatenate to get the input token , and then embed the image into features and dense embedding Concatenate to get the input embedding of the Transformer module ; Enter the token and input embedding Input Transformer module to get updated input embedding , and then embed the updated input Input to two deconvolution Transposed Conv for upsampling to get image feature embedding , and then the image features are embedded Input into the multi-layer perceptron MLP, according to the mask token embedding Embedding image features Perform segmentation. Finally, MLP outputs the predicted segmentation results. .

[0008] S3 is as follows: According to the segmentation results predicted by the model and the actual segmentation results, pixel-based, region-based, distribution-based and shape-based losses are calculated, and a multi-task balancing mechanism MBM is introduced to dynamically adjust the weights of pixel-based, region-based, distribution-based and shape-based losses. An adaptive sharpness norm minimization ASNM strategy is also introduced to seek the flat minimum of the model by optimizing the maximum loss and gradient norm, thereby improving and optimizing the lightweight medical image segmentation model, selecting the optimal parameters to update the model, and obtaining a lightweight medical image segmentation model after parameter update; The pixel-based MSE loss is calculated by calculating the predicted segmentation result. The prediction accuracy of the lightweight medical image segmentation model is quantified by comparing each pixel value in the image with the pixel value corresponding to the true label. The calculation formula is as follows: , in, represents the pixel-based MSE loss, Represents the predicted segmentation result The total number of pixels in the image, express The index of Indicates The predicted pixel values, Indicates The pixel value of the true label; The region-based DC loss emphasizes the predicted segmentation results. The overlapping area between the real label and the real label is used to maintain the accuracy of the shape and boundary of the segmented area. The calculation formula is as follows: , in, represents the DC loss based on the area; The distribution-based CE loss ensures segmentation accuracy by accurately classifying individual pixels. The calculation formula is as follows: , in, represents the distribution-based CE loss; Shape-based SD loss passes the predicted segmentation result The difference between the mask distance transform map and the image is used to enhance the shape features of the segmented target. The calculation formula is as follows: , in, represents the shape-based SD loss, Indicates the total number of channels, express The index of Represents the predicted segmentation result The distance transformation map with the mask Pixels between channels The difference between The pixel-based, region-based, distribution-based, and shape-based losses are combined into a resource framework to define MBM, which is calculated as follows: , in, represents pixel-based, region-based, distribution-based, and shape-based losses, represents the number of losses, , represents the pixel-based MSE loss, Based on the DC loss of the area, Distribution-based CE loss, Shape-based SD loss, Represents the learnable parameters corresponding to each loss.

[0009] The adaptive sharpness norm minimization ASNM strategy is as follows: Determine the training distribution of image space and mask space based on the data in the training set , with a point Center Determine the open sphere for the radius , represents any point in the open sphere, and the open sphere is represented by ,||.|| represents the Euclidean norm; The overall loss of the sample space is , the empirical loss function of the sample space is , Express expectations, represents the data distribution, represents the average loss of all samples in the sample space, Represents a pixel, Indicates pixels, Indicates the total number of pixels; Calculate the empirical loss function exist The first-order derivative and Hessian matrix of the point, and Represent the empirical loss function respectively exist First-order derivatives and Hessian matrices of points; The optimization process of the ASNM strategy includes two steps, as follows: Step 1: Finding Flat Minima For each small batch of data, the model parameters are adjusted to maximize the loss. ASNM finds The perturbation that maximizes the loss , so that the model parameters are adjusted towards the non-steep area of ​​the loss landscape, and then the model parameters are updated. The calculation formula is as follows: , in, represents the overall loss, represents disturbance, represents the search for a non-steep parameter configuration in the loss, It means finding the perturbation that maximizes the loss ; Experience loss function exist The first-order flatness at a point is defined as follows: , in, express exist First-order flatness at the point, represents the radius of the open sphere, Indicated in The first derivative of a point, It means taking the maximum point in the open ball; Step 2: Approximate The optimization is performed by calculating the first-order flatness at the point The gradient at the point optimizes the maximum eigenvalue of the Hessian matrix, making the curvature at the minimum point in the perturbation field smoother. The calculation formula is as follows: ,in, , in, represents the derivative of first-order flatness, Representation function The first derivative of Indicates gradient adjustment of the loss function.

[0010] The present invention also provides a lightweight multimodal medical image segmentation system, which implements a lightweight multimodal medical image segmentation method, including the following modules: Data acquisition module: collects medical images of various imaging modes to build data sets; Data preprocessing module: preprocess the data set in the data acquisition module and divide it into training set and test set; Lightweight medical image segmentation module: input the training set in the data domain processing module into the lightweight medical image segmentation module, train the constructed lightweight medical image segmentation model, and obtain the predicted segmentation result; Optimization module: Based on the loss function and ASNM strategy, the lightweight medical image segmentation model is improved and optimized to obtain the lightweight medical image segmentation model with updated parameters. The test set in the data preprocessing module is input into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

[0011] The effects provided in the content of the invention are only the effects of the embodiments, rather than all the effects of the invention. The above technical solution has the following advantages or beneficial effects: The present invention proposes a lightweight multimodal medical image segmentation method and system to reduce resource dependence and improve model segmentation accuracy and generalization performance; LViT in the present invention is a scaled-down visual converter using a progressive shrinkage method. By using LViT to build an image encoder, the model parameters can be significantly reduced. In addition, the present invention ensures the segmentation accuracy of the lightweight model by introducing feature knowledge distillation, extracts knowledge from a well-trained MedSAM using a teacher-student framework, and transfers the knowledge of the teacher model encoder to the LViT encoder, effectively transferring the rich features and decision information in the teacher model to the student model; The present invention also strengthens the focus on high-dimensional features through a multi-task balancing mechanism, enhances the model's learning ability for difficult tasks, reduces the uncertainty of prediction results, and captures more detailed information through high-dimensional features; in order to prevent the model from falling into a sharp minimum point in the loss space, the present invention introduces an adaptive sharpness norm minimization ASNM strategy, which seeks the flat minimum of the model by optimizing the maximum loss and gradient norm, so that the model has excellent generalization ability while capturing detailed information. It aims to enable the model to comprehensively learn the diversity of data and improve the generalization performance of the model by paying attention to the flatness of the loss function and the normative processing of the gradient.

[0012] In summary, the present invention can help doctors quickly identify and locate lesions in a clinical environment by optimizing computing efficiency and resource utilization, thereby improving work efficiency and ultimately improving the treatment and prognosis of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0014] Figure 1 It is a schematic diagram of the overall framework of the method of the present invention.

[0015] Figure 2 This is a diagram of the experimental results of the present invention.

[0016] Figure 3 The figure is a comparison chart of the experimental results of the method of the present invention and the existing method. DETAILED DESCRIPTION

[0017] In order to clearly illustrate the technical features of the present invention, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0018] Example 1 like Figure 1 As shown, the present invention proposes a lightweight multimodal medical image segmentation method, which is as follows: S1. Collect medical images of various imaging modes to build a data set, preprocess the images in the data set, and then divide the preprocessed data set into a training set and a test set; S2. Construct a lightweight medical image segmentation model, which includes an image encoder, a hint encoder, and a mask decoder. Input the medical images in the training set into the lightweight medical image segmentation model to perform medical image segmentation and obtain the segmentation results predicted by the model. S3, improving and optimizing the lightweight medical image segmentation model to obtain a lightweight medical image segmentation model with updated parameters; S4. Input the medical images in the test set into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

[0019] S1 is as follows: S1.1. Constructing the dataset: The medical images collected in the dataset are all from the CVPR challenge. The data modalities in the CVPR challenge include CT, MRI, X-Ray, Endoscopy, US, and Fundus images. Experts manually label the tissues, organs, and cancer types in the images accurately. S1.2, Data preprocessing: Convert 3D medical imaging data into a series of 2D images; Data augmentation is performed on all modal data, and the data augmentation operation is horizontal and vertical flipping with a probability of 50%; Resize and pad the image and label to 256x256 size. Normalize the image and adjust the intensity value to the range of (0,1); S1.3. Divide the data set: The preprocessed data set is divided into training sets in a ratio of 4:1. and test set , , Indicates the number of data modal types, Indicates The sub-datasets of the data modalities in the data set are each sub-dataset of a modality, and each sub-dataset of a modality contains a different number of image data of the corresponding modality.

[0020] S2 is as follows: S2.1. Construct a lightweight medical image segmentation model, which includes an image encoder, a hint encoder, and a mask decoder; (1) Image encoder: The image encoder includes 12 lightweight visual Transformer modules LViT in sequence. The 12 LViTs are divided into 4 stages. The first stage includes 2 LViTs, the second stage includes 2 LViTs, the third stage includes 6 LViTs, and the fourth stage includes 2 LViTs. Each LViT includes layer normalization Layer Norm, multi-head self-attention, layer normalization Layer Norm and multi-layer perceptron MLP in sequence. (2) Hint encoder: The hint encoder consists of a position embedding layer and a mask downsampling module. The position embedding layer calls To calculate the embedding of the bounding box, the mask downsampling module includes the convolution layer Conv, the layer normalization Layer Norm, the activation function act, the convolution layer Conv, the layer normalization Layer Norm, the activation function act and the convolution layer Conv in sequence; (3) Mask decoder: The mask decoder includes feature fusion module, Transformer module, Transformer module, deconvolution Transposed Conv, deconvolution Transposed Conv and multi-layer perceptron MLP in sequence; S2.2. Image data in the training set Input to the lightweight medical image segmentation model, Represents the training set Sub-dataset Any image data in Include images and mask , image data Image feature embedding is obtained after processing by the image encoder , introduce teacher model and feature knowledge distillation technology to embed image features Align and get image feature embedding , image data After processing by the hint decoder, the sparse embedding is obtained and dense embedding , embedding image features into , sparse embedding and dense embedding Input to the mask decoder, and the predicted segmentation result is obtained after processing by the mask decoder.

[0021] The image segmentation process of the lightweight medical image segmentation model is as follows: S2.2.1. Image data in the training set Input to the image encoder, the image encoder reads the image data Images in ,image After four stages of LViT, the final output image feature embedding ; Assume that the input data of LViT is , the processing process of each LViT on the input data is First, it passes through the first Layer Norm, then the first Layer Norm output is input into Multi-head Self-Attention, and then the output of Multi-head Self-Attention is combined with Fusion, get ,Will Input to the second LayerNorm, and then input the output of the second LayerNorm into the MLP, and compare the output of the MLP with Fusion is performed to obtain the final output of LViT ; , , MedSAM, an artificial intelligence model specifically for medical image segmentation, is introduced. MedSAM includes 12 Transformer modules in sequence. The 12 Transformer modules are divided into four stages. Each stage includes 2, 2, 6, and 2 Transformer modules in sequence. Input into MedSAM to get image feature embedding ; MedSAM is used as the teacher model, the image encoder is used as the student model, and then the feature knowledge distillation technology is introduced to embed the image features Embedding based on image features Align and get the aligned image feature embedding ; During the feature knowledge distillation process, the teacher model is supervised and optimized through a combination of binary cross-entropy loss and dice loss. The loss function aligns the image feature embeddings of the teacher model and the student model, and the objective function of knowledge distillation training The details are as follows: ; S2.2.2. The image data in the training set Input to prompt encoder, prompt encoder to read image data The mask in , prompting the encoder to read the mask Generate a hint bounding box, which is composed of two corner point coordinates. and , and Represent the horizontal and vertical coordinates of the two corner points respectively, and then input the coordinates of the two corner points of the prompt bounding box into the boundary embedding module embed_boxes, adjust the coordinates to the pixel center, and encode the positions of the two corner points through the position embedding layer to generate the same image embedding features. Aligned Sparse Embedding , the calculation formula is as follows: , , in, represents the operation of computing bounding box embeddings, represents boundary embedding, Represents a splicing operation; At the same time, the mask Through the mask downsampling module, after multiple convolutional layers of downsampling, the generated image embedding features are generated. Aligned Dense Embedding , the calculation formula is as follows: , in, Represents the activation function operation; S2.2.3. Embedding features into images for each image in the training set , sparse embedding and dense embedding Input to the decoder masker, first embed the hint bounding box IOU token in the feature fusion module and mask token embedding Splice to get fusion features , and then sparse embedding Re- and fusion features Concatenate to get the input token , and then embed the image into features and dense embedding Concatenate to get the input embedding of the Transformer module ; Enter the token and input embedding Input Transformer module to get updated input embedding , and then embed the updated input Input to two deconvolution Transposed Conv for upsampling to get image feature embedding , and then the image features are embedded Input into the multi-layer perceptron MLP, according to the mask token embedding Embedding image features Perform segmentation. Finally, MLP outputs the predicted segmentation results. .

[0022] S3 is as follows: According to the segmentation results predicted by the model and the actual segmentation results, pixel-based, region-based, distribution-based and shape-based losses are calculated, and a multi-task balancing mechanism MBM is introduced to dynamically adjust the weights of pixel-based, region-based, distribution-based and shape-based losses. An adaptive sharpness norm minimization ASNM strategy is also introduced to seek the flat minimum of the model by optimizing the maximum loss and gradient norm, thereby improving and optimizing the lightweight medical image segmentation model, selecting the optimal parameters to update the model, and obtaining a lightweight medical image segmentation model after parameter update; The pixel-based MSE loss is calculated by calculating the predicted segmentation result. The prediction accuracy of the lightweight medical image segmentation model is quantified by comparing each pixel value in the image with the pixel value corresponding to the true label. The calculation formula is as follows: , in, represents the pixel-based MSE loss, Represents the predicted segmentation result The total number of pixels in the image, express The index of Indicates The predicted pixel values ​​are Indicates The pixel value of the true label; The region-based DC loss emphasizes the predicted segmentation results. The overlapping area between the real label and the real label is used to maintain the accuracy of the shape and boundary of the segmented area. The calculation formula is as follows: , in, represents the DC loss based on the area; The distribution-based CE loss ensures segmentation accuracy by accurately classifying individual pixels. The calculation formula is as follows: , in, represents the distribution-based CE loss; Shape-based SD loss passes the predicted segmentation result The difference between the mask distance transform map and the image is used to enhance the shape features of the segmented target. The calculation formula is as follows: , in, represents the shape-based SD loss, Indicates the total number of channels, express The index of Represents the predicted segmentation result The distance transformation map with the mask Pixels between channels The difference between The pixel-based, region-based, distribution-based, and shape-based losses are combined into a resource framework to define MBM, which is calculated as follows: , in, represents pixel-based, region-based, distribution-based, and shape-based losses, represents the number of losses, , represents the pixel-based MSE loss, Based on the DC loss of the area, Distribution-based CE loss, Shape-based SD loss, Represents the learnable parameters corresponding to each loss.

[0023] The adaptive sharpness norm minimization ASNM strategy is as follows: Determine the training distribution of image space and mask space based on the data in the training set , with a point Center Determine the open sphere for the radius , represents any point in the open sphere, and the open sphere is represented by ,||.|| represents the Euclidean norm; The overall loss of the sample space is , the empirical loss function of the sample space is , Express expectations, represents the data distribution, represents the average loss of all samples in the sample space, Represents a pixel, Indicates pixels, Indicates the total number of pixels; Calculate the empirical loss function exist The first-order derivative and Hessian matrix of the point, and Represent the empirical loss function respectively exist First-order derivatives and Hessian matrices of points; The optimization process of the ASNM strategy includes two steps, as follows: Step 1: Finding Flat Minima For each small batch of data, the model parameters are adjusted to maximize the loss. ASNM finds The perturbation that maximizes the loss , so that the model parameters are adjusted towards the non-steep area of ​​the loss landscape, and then the model parameters are updated. The calculation formula is as follows: , in, represents the overall loss, represents disturbance, represents the search for a non-steep parameter configuration in the loss, It means finding the perturbation that maximizes the loss ; Experience loss function exist The first-order flatness at a point is defined as follows: , in, express exist First-order flatness at the point, represents the radius of the open sphere, Indicated in The first derivative of a point, It means taking the maximum point in the open ball; Step 2: Approximate The optimization is performed by calculating the first-order flatness at the point The gradient at the point optimizes the maximum eigenvalue of the Hessian matrix, making the curvature at the minimum point in the perturbation field smoother. The calculation formula is as follows: , in, , in, represents the derivative of first-order flatness, Representation function The first derivative of Indicates gradient adjustment of the loss function.

[0024] Example 2 The present invention also provides a lightweight multimodal medical image segmentation system, which implements a lightweight multimodal medical image segmentation method, including the following modules: Data acquisition module: collects medical images of various imaging modes to build data sets; Data preprocessing module: preprocess the data set in the data acquisition module and divide it into training set and test set; Lightweight medical image segmentation module: input the training set in the data domain processing module into the lightweight medical image segmentation module, train the constructed lightweight medical image segmentation model, and obtain the predicted segmentation result; Optimization module: Based on the loss function and ASNM strategy, the lightweight medical image segmentation model is improved and optimized to obtain the lightweight medical image segmentation model with updated parameters. The test set in the data preprocessing module is input into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

[0025] Example 3 like Figure 2 As shown, the three groups of images are abdominal CT, intestinal polyps and chest X-Ray images, respectively. The images from left to right in each group are input, label and prediction results respectively. The input images in the three groups of images are processed by the method of the present invention. The first group of abdominal CT predicts the liver, spleen, pancreas, left and right kidneys, the second group predicts intestinal polyps, and the third group predicts the left and right lungs.

[0026] Depend on Figure 2It can be seen from the prediction results that the network model in the present invention performs satisfactorily in the automatic cross-modal target segmentation, showing strong stability and consistency, and a low mis-segmentation rate. The segmentation accuracy of the network model in the present invention is very high, and it can accurately identify and segment the region of interest. For the precise segmentation of the target task, the model also shows good general performance and can effectively perform cross-modal segmentation of the region of interest. This shows that although the cross-modal segmentation of medical images is more challenging, the network model in the present invention can still provide reliable prediction results and show high stability and accuracy.

[0027] Example 4 In order to prove that the present invention is superior to the existing method, Figure 2 The input images in the three groups of pictures are processed by the method of the present invention and the existing method (U-Mamba represents a general network for biomedical image segmentation, nnUNet represents a deep learning model based on the U-Net architecture, nnFormer represents a 3D medical image segmentation network based on a cross Transformer structure, SwinUNETR represents a combination of window Transformer and U-Net, and U-MedSAM represents an arbitrary medical image segmentation network based on uncertainty) to obtain the following: Figure 3 The segmentation result shown is Figure 3 The experimental results show that the method of the present invention performs well in detail processing and can completely and accurately segment the target organ. Even in the second set of contrast images with poor contrast, the method of the present invention can still maintain accurate segmentation capabilities, while other models perform poorly in capturing complex boundary features. Figure 3 The yellow arrow in the middle indicates that the prediction result does not match the lesion area, showing obvious under-segmentation. At the same time, the method of the present invention is compared with the existing method in terms of dice similarity coefficient and surface normalized distance. The dice similarity coefficient is used to evaluate the degree of regional overlap between the expert-annotated mask and the segmentation result. The larger the value, the higher the degree of overlap. The surface normalized distance mainly evaluates the boundary consistency between the expert-annotated mask and the segmentation result. It can be seen from Table 1 that compared with other existing methods, the method of the present invention can be optimized and improved to adapt to different image features, accurately capture subtle structural changes in medical images, and thus significantly improve segmentation accuracy. Table 1 Comparison results between the method of the present invention and the existing method In summary, it can be proved that the lightweight multimodal medical image segmentation framework proposed in the present invention has significant advantages over the existing technology; first, through the bounding box prompt and knowledge distillation technology, the model parameters are greatly reduced, thereby reducing the computational complexity, which enables it to run efficiently in a resource-constrained environment; in addition, the framework is flexible and adaptable, and adopts a multi-task balancing mechanism to dynamically adjust the model learning resources, which can effectively respond to changes in demand according to the priorities of different tasks, and further optimize the segmentation effect; the introduction of the ASNM training optimization strategy also accelerates the convergence speed of the model, enabling it to quickly adapt to different types of medical images, thereby improving the training efficiency; finally, the lightweight and efficient design makes the deployment of the model in practical applications more convenient, and suitable for rapid application in clinical environments; in summary, these advantages make the method in the present invention show important application potential and practical value in the field of medical image segmentation.

[0028] Although the above describes the specific implementation mode of the invention in conjunction with the drawings, it is not intended to limit the scope of protection of the invention. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present invention.

Claims

1. A lightweight multimodal medical image segmentation method, characterized in that: The following steps are involved: S1. Collect medical images of various imaging modes to build a data set, preprocess the images in the data set, and then divide the preprocessed data set into a training set and a test set; S2. Construct a lightweight medical image segmentation model, which includes an image encoder, a prompt encoder, and a mask decoder. Input the medical images in the training set into the lightweight medical image segmentation model to perform medical image segmentation and obtain the segmentation results predicted by the model. S3, improving and optimizing the lightweight medical image segmentation model to obtain a lightweight medical image segmentation model with updated parameters; S4. Input the medical images in the test set into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

2. The lightweight multimodal medical image segmentation method according to claim 1, characterized in that: S1 is as follows: S1.

1. Constructing the dataset: The medical images collected in the dataset are all from the CVPR challenge. The data modalities in the CVPR challenge include CT, MRI, X-Ray, Endoscopy, US, and Fundus images. Experts manually annotate the tissues, organs, and cancer types in the images with precise labels. S1.2, Data preprocessing: Convert 3D medical imaging data into a series of 2D images; Data augmentation is performed on all modal data, and the data augmentation operation is horizontal and vertical flipping with a probability of 50%; Resize and pad the image and label to 256x256 size. Normalize the image and adjust the intensity value to the range of (0,1); S1.

3. Divide the data set: The preprocessed data set is divided into training sets in a ratio of 4:

1. and test set , , Indicates the number of data modal types, Indicates The sub-datasets of the data modalities in the data set are each sub-dataset of a modality, and each sub-dataset of a modality contains a different number of image data of the corresponding modality.

3. The lightweight multimodal medical image segmentation method according to claim 2, characterized in that: S2 is as follows: S2.

1. Construct a lightweight medical image segmentation model, which includes an image encoder, a hint encoder, and a mask decoder; (1) Image encoder: The image encoder includes 12 lightweight visual Transformer modules LViT in sequence. The 12 LViTs are divided into 4 stages. The first stage includes 2 LViTs, the second stage includes 2 LViTs, the third stage includes 6 LViTs, and the fourth stage includes 2 LViTs. Each LViT includes layer normalization Layer Norm, multi-head self-attention, layer normalization Layer Norm and multi-layer perceptron MLP in sequence. (2) Hint encoder: The hint encoder consists of a position embedding layer and a mask downsampling module. The position embedding layer calls To calculate the embedding of the bounding box, the mask downsampling module includes the convolution layer Conv, the layer normalization Layer Norm, the activation function act, the convolution layer Conv, the layer normalization Layer Norm, the activation function act and the convolution layer Conv in sequence; (3) Mask decoder: The mask decoder includes feature fusion module, Transformer module, Transformer module, deconvolution Transposed Conv, deconvolution Transposed Conv and multi-layer perceptron MLP in sequence; S2.

2. Image data in the training set Input to the lightweight medical image segmentation model, Represents the training set Sub-dataset Any image data in Include images and mask , image data Image feature embedding is obtained after processing by the image encoder , introduce teacher model and feature knowledge distillation technology to embed image features Align and get image feature embedding , image data After processing by the hint decoder, the sparse embedding is obtained and dense embedding , embedding image features into , sparse embedding and dense embedding Input to the mask decoder, and the predicted segmentation result is obtained after processing by the mask decoder.

4. The lightweight multimodal medical image segmentation method according to claim 3, characterized in that: The image segmentation process of the lightweight medical image segmentation model is as follows: S2.2.

1. Image data in the training set Input to the image encoder, the image encoder reads the image data Images in ,image After four stages of LViT, the final output image feature embedding ; Assume that the input data of LViT is , the processing process of each LViT on the input data is First, it passes through the first Layer Norm, then the first Layer Norm output is input into Multi-head Self-Attention, and then the output of Multi-head Self-Attention is combined with Fusion, get ,Will Input to the second LayerNorm, and then input the output of the second LayerNorm into the MLP, and compare the output of the MLP with Fusion is performed to obtain the final output of LViT ; ， ， MedSAM, an artificial intelligence model specifically for medical image segmentation, is introduced. MedSAM includes 12 Transformer modules in sequence. The 12 Transformer modules are divided into four stages. Each stage includes 2, 2, 6, and 2 Transformer modules in sequence. Input into MedSAM to get image feature embedding ; MedSAM is used as the teacher model, the image encoder is used as the student model, and then the feature knowledge distillation technology is introduced to embed the image features Embedding based on image features Align and get the aligned image feature embedding ; During the feature knowledge distillation process, the teacher model is supervised and optimized through a combination of binary cross-entropy loss and dice loss. The loss function aligns the image feature embeddings of the teacher model and the student model, and the objective function of knowledge distillation training The details are as follows: ； S2.2.

2. The image data in the training set Input to prompt encoder, prompt encoder to read image data The mask in , prompting the encoder to read the mask Generate a hint bounding box, which is composed of two corner point coordinates. and , and Represent the horizontal and vertical coordinates of the two corner points respectively, and then input the coordinates of the two corner points of the prompt bounding box into the boundary embedding module embed_boxes, adjust the coordinates to the pixel center, and encode the positions of the two corner points through the position embedding layer to generate the same image embedding features. Aligned Sparse Embedding , the calculation formula is as follows: ， ， in, represents the operation of computing bounding box embeddings, represents boundary embedding, Represents a splicing operation; At the same time, the mask Through the mask downsampling module, after multiple convolutional layers of downsampling, the generated image embedding features are generated. Aligned Dense Embedding , the calculation formula is as follows: ， in, Represents the activation function operation; S2.2.

3. Embedding features into images for each image in the training set , sparse embedding and dense embedding Input to the decoder masker, first embed the hint bounding box IOU token in the feature fusion module and mask token embedding Splice to get fusion features , and then sparse embedding Re- and fusion features Concatenate to get the input token , and then embed the image into features and dense embedding Concatenate to get the input embedding of the Transformer module ; Enter the token and input embedding Input Transformer module to get updated input embedding , and then embed the updated input Input to two deconvolution Transposed Conv for upsampling to get image feature embedding , and then the image features are embedded Input into the multi-layer perceptron MLP, according to the mask token embedding Embedding image features Perform segmentation. Finally, MLP outputs the predicted segmentation results. .

5. The lightweight multimodal medical image segmentation method according to claim 4, characterized in that S3 The details are as follows: According to the segmentation results predicted by the model and the actual segmentation results, pixel-based, region-based, distribution-based and shape-based losses are calculated, and a multi-task balancing mechanism MBM is introduced to dynamically adjust the weights of pixel-based, region-based, distribution-based and shape-based losses. An adaptive sharpness norm minimization ASNM strategy is also introduced to seek the flat minimum of the model by optimizing the maximum loss and gradient norm, thereby improving and optimizing the lightweight medical image segmentation model, selecting the optimal parameters to update the model, and obtaining a lightweight medical image segmentation model after parameter update; The pixel-based MSE loss is calculated by calculating the predicted segmentation result. The prediction accuracy of the lightweight medical image segmentation model is quantified by comparing each pixel value in the image with the pixel value corresponding to the true label. The calculation formula is as follows: ， in, represents the pixel-based MSE loss, Represents the predicted segmentation result The total number of pixels in the image, express The index of Indicates The predicted pixel values ​​are Indicates The pixel value of the true label; The region-based DC loss emphasizes the predicted segmentation results. The overlapping area between the real label and the real label is used to maintain the accuracy of the shape and boundary of the segmented area. The calculation formula is as follows: ， in, represents the DC loss based on the area; The distribution-based CE loss ensures segmentation accuracy by accurately classifying individual pixels. The calculation formula is as follows: ， in, represents the distribution-based CE loss; Shape-based SD loss passes the predicted segmentation result The difference between the distance transform map and the mask is used to enhance the shape characteristics of the segmented target. The calculation formula is as follows: ， in, represents the shape-based SD loss, Indicates the total number of channels, express The index of Represents the predicted segmentation result The distance transformation map with the mask Pixels between channels The difference between The pixel-based, region-based, distribution-based, and shape-based losses are combined into a resource framework to define MBM, which is calculated as follows: ， in, represents pixel-based, region-based, distribution-based, and shape-based losses, represents the number of losses, , represents the pixel-based MSE loss, Based on the DC loss of the area, Distribution-based CE loss, Shape-based SD loss, Represents the learnable parameters corresponding to each loss.

6. The lightweight multimodal medical image segmentation method according to claim 5, characterized in that: The specific ASNM strategy for minimizing the sharpness norm is as follows: Determine the training distribution of image space and mask space based on the data in the training set , with a point Center Determine the open sphere for the radius , represents any point in the open sphere, and the open sphere is represented by ,||.|| represents the Euclidean norm; The overall loss of the sample space is , the empirical loss function of the sample space is , Express expectations, represents the data distribution, represents the average loss of all samples in the sample space, Represents a pixel, Indicates pixels, Indicates the total number of pixels; Calculate the empirical loss function exist The first-order derivative and Hessian matrix of the point, and Represent the empirical loss function respectively exist First-order derivatives and Hessian matrices of points; The optimization process of the ASNM strategy includes two steps, as follows: Step 1: Finding Flat Minima For each small batch of data, the model parameters are adjusted to maximize the loss. ASNM finds The perturbation that maximizes the loss , so that the model parameters are adjusted towards the non-steep area of ​​the loss landscape, and then the model parameters are updated. The calculation formula is as follows: ， in, represents the overall loss, represents disturbance, represents the search for a non-steep parameter configuration in the loss, It means finding the perturbation that maximizes the loss ; Experience loss function exist The first-order flatness at a point is defined as follows: ， in, express exist First-order flatness at the point, represents the radius of the open sphere, Indicated in The first derivative of a point, It means taking the maximum point in the open ball; Step 2: Approximate The optimization is performed by calculating the first-order flatness at the point The gradient at the point optimizes the maximum eigenvalue of the Hessian matrix, making the curvature at the minimum point in the perturbation field smoother. The calculation formula is as follows: ， in, ， in, represents the derivative of first-order flatness, Representation function The first derivative of Indicates gradient adjustment of the loss function.

7. A lightweight multimodal medical image segmentation system, executing a lightweight multimodal medical image segmentation method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Data acquisition module: collects medical images of various imaging modes to build data sets; Data preprocessing module: preprocess the data set in the data acquisition module and divide it into training set and test set; Lightweight medical image segmentation module: input the training set in the data domain processing module into the lightweight medical image segmentation module, train the constructed lightweight medical image segmentation model, and obtain the predicted segmentation result; Optimization module: Based on the loss function and ASNM strategy, the lightweight medical image segmentation model is improved and optimized to obtain the lightweight medical image segmentation model with updated parameters. The test set in the data preprocessing module is input into the lightweight medical image segmentation model with updated parameters to obtain the final predicted segmentation result.

Images