Few-sample medical image segmentation method combining CNN and SAM

Abstract

The invention belongs to the technical field of medical image analysis, and particularly relates to a few-sample medical image segmentation method combining CNN and SAM, comprising: constructing an SAM model for a few-sample medical image combining CNN and SAM, the model comprising a CNN feature extraction module, a few-sample fine tuning module and a generation prompt module; training the SAM model by using cross entropy loss, Dice loss and MSE loss to obtain a trained SAM model; and the SAM segmentation module is used for generating a prompt module, and guiding the model to carry out accurate target region segmentation by using a small number of labeled samples or by clicking a prompt mode. According to the invention, the CNN feature extraction module is introduced to carry out feature extraction on local information, missing local information is supplemented for an SAM network, the few-sample fine tuning module is introduced to migrate image information to the field of medical image learning, accurate segmentation on a medical image is realized, and the generation prompt module is introduced to avoid manual interaction and generate point prompt. And the diagnosis accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image analysis, and in particular to a few-sample medical image segmentation method combining CNN and SAM. Background Art

[0002] Medical image segmentation tasks usually face problems such as scarcity of labeled samples, high labeling costs, and data imbalance. The labeling process of medical images often relies on professional medical experts, which not only requires experts to have deep professional knowledge, but also requires a lot of time and effort. Because the labeling process is cumbersome and time-consuming, it becomes difficult to obtain large-scale high-quality labeled data, which makes the model lack sufficient labeled samples during training, which to a certain extent limits the effectiveness and popularity of deep learning methods. Secondly, the problem of data imbalance in medical images is particularly prominent. In many medical imaging datasets, cases of certain diseases are relatively rare (such as rare diseases or early lesions), while samples of healthy or other common diseases dominate. This data imbalance problem will cause the segmentation model to tend to identify more common categories in the data during training, thereby affecting the recognition accuracy of rare diseases or subtle lesions.

[0003] Segment Anything Model (SAM) uses advanced deep learning algorithms and in-depth analysis of massive natural image data to play an important role in natural image segmentation. With its powerful zero-sample learning ability, SAM can automatically identify and accurately segment various objects or scenes in images without a large number of labeled samples. Combined with large-scale pre-training data, SAM can achieve fast and accurate segmentation. By combining the computing power of artificial intelligence with image processing technology, SAM not only improves segmentation efficiency, but also enhances the accuracy of complex object boundaries.

[0004] Although SAM has the ability of zero-shot learning, its performance is still limited in few-shot learning tasks. In medical image segmentation tasks, especially when there are only (a small number of) annotated samples, SAM may have difficulty in fully capturing the features in the image, especially when the lesions or complex structures in the data are not representative enough. SAM achieves zero-shot learning through large-scale pre-trained models, but the source of pre-training datasets is mainly natural images, not medical images. Therefore, SAM may not be able to fully adapt to the special needs of the medical field in medical image segmentation tasks.

[0005] To address these challenges, the fine-tuning pre-trained SAM method is widely used. SAM is trained on a large-scale general image dataset in the pre-training stage, while fine-tuning allows SAM to be optimized for specific fields (such as medical images). By fine-tuning on medical imaging datasets, the model can better learn the unique features in medical images, such as anatomical structures, morphology of lesion areas, etc., significantly improving the performance in medical image segmentation. A major advantage of fine-tuning a large model is that it can still achieve good performance under few sample conditions. Traditional deep learning models usually require a large amount of labeled data for training, while fine-tuning can make full use of existing pre-trained models and further optimize them with a small number of labeled samples. The fine-tuned SAM large model can learn more targeted features on medical image datasets in specific fields, which enables the model to better adapt and generalize when processing images from different patients and different imaging conditions.

[0006] Convolutional neural network is a core architecture in deep learning and is widely used in image processing tasks, especially in image segmentation. CNN can effectively extract spatial hierarchical features in images and is particularly suitable for processing image data with local structure and spatial correlation. SAM combines CNN technology to organically combine SAM's zero-sample learning ability with CNN's powerful ability in image feature extraction to improve performance in medical image segmentation, natural image segmentation and other fields. Summary of the invention

[0007] 1. Technical issues to be resolved In view of the shortcomings of the prior art, the present invention provides a few-sample medical image segmentation method combining CNN and SAM. By combining convolutional neural network and Segment Anything Model, a novel few-sample medical image segmentation method is designed, and a medical image segmentation strategy based on deep learning and few-sample learning is proposed. It can achieve accurate segmentation of target areas in medical images with a small number of labeled samples, and can effectively extract complex features in medical images. At the same time, with the help of SAM's optimized segmentation mechanism, it reduces the demand for a large amount of labeled data, which not only improves the segmentation accuracy, but also maintains efficient performance in complex scenarios such as data imbalance and cross-modal adaptation, and solves the problems raised in the above background technology.

[0008] (II) Technical solution In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: A few-sample medical image segmentation method combining CNN and SAM, comprising: Construction steps: Construct a SAM model for few-sample medical images combining CNN and SAM. The model includes a CNN feature extraction module, a few-sample fine-tuning module, and a prompt generation module. The CNN feature extraction module uses a convolutional neural network to extract features from preprocessed medical images. CNN automatically learns deep spatial features from medical images through multi-layer convolution and pooling operations, especially the extraction of complex structures; the few-sample fine-tuning module uses a small number of labeled samples for fine-tuning in the few-sample learning scenario. By utilizing the powerful transfer learning ability of SAM, a small amount of data is optimized based on the existing pre-trained model, so that the model can effectively learn and segment when labeled samples are scarce; the prompt generation module automatically generates prompts for guiding segmentation by analyzing the content of the image. By parsing the visual information of the image, the model can identify potential segmentation objects or areas and generate corresponding point prompts, which are then used to guide the SAM model for accurate segmentation.

[0009] Training step: using cross entropy loss, Dice loss and MSE loss to train the SAM model to obtain a trained SAM model, wherein the parameters of the image encoder are frozen during the training process to improve training efficiency and reduce computing resource requirements; Segmentation steps: First, the medical image is preprocessed to obtain the processed medical image, and the processed image is input into the CNN feature extraction module and the SAM segmentation module respectively. The CNN module automatically extracts local features in the medical image through the convolution layer and focuses on the detailed information of the key areas in the image; at the same time, the SAM segmentation module generates a prompt module, uses a small number of labeled samples or guides the model to perform accurate target area segmentation by clicking on prompts. SAM optimizes the segmentation result according to the input prompt information, and then performs domain migration through the few-sample fine-tuning module, and combines it with the features extracted by CNN to finally output accurate medical image segmentation results.

[0010] (III) Beneficial effects Compared with the prior art, the present invention provides a few-sample medical image segmentation method combining CNN and SAM, which has the following beneficial effects: 1. The present invention integrates an efficient CNN feature extraction module. The model does not need to change the original SAM structure and can effectively handle the medical image segmentation task under the condition of few samples. It does not need a SAM with a large number of labeled samples. It can automatically extract deep features in the image through the CNN module and still achieve high-precision image segmentation with a small number of labeled samples. Through the CNN feature extraction module, the model can learn the key local information of the image from limited data and provide richer feature representation for subsequent segmentation tasks, which greatly improves the accuracy and robustness of few-sample medical image segmentation.

[0011] 2. The present invention integrates a few-sample fine-tuning module to significantly improve the performance of the SAM model while ensuring segmentation accuracy and training efficiency. Compared with training the entire model from scratch or using a larger labeled data set, the few-sample fine-tuning module can quickly optimize the parameters of the SAM model with only a small number of labeled samples, effectively improving the segmentation ability of the model under few-sample conditions. This module not only saves annotation and computing resources, but also greatly speeds up the training speed of the model, while ensuring high accuracy and robustness in medical image segmentation tasks.

[0012] 3. The present invention integrates a prompt generation module. The segmentation accuracy and robustness of SAM under the condition of few samples are significantly enhanced. It can effectively use a small number of labeled samples and automatically generate prompts for guiding segmentation by analyzing the content of the image. By parsing the visual information of the image, the model can identify potential segmentation objects or regions and generate corresponding point prompts, which are then used to guide the SAM model to perform accurate segmentation. This module enables SAM to accurately process complex structures in medical images even when labeled data is scarce, significantly improving the practicality and reliability of medical image segmentation in clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 It is a schematic diagram of the SAM network structure of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] like Figure 1 As shown, an embodiment of the present invention proposes: a few-sample medical image segmentation method combining CNN and SAM, including an end-to-end process from image preprocessing to prompting to generate a final segmentation result; First, the medical images to be segmented are extracted from the data sets of different modalities and preprocessed. Next, the images are processed by the parallel CNN feature extraction module and SAM segmentation module. In the CNN module, local features in medical images are automatically extracted through convolutional neural networks, especially for key areas (such as tumors, organs, etc.) to perform deep feature learning and generate feature representations of images. The formula for CNN feature extraction is as follows:

[0016] in, is a convolution operation, applied On the input, it is used to extract local information in images and feature maps. It represents a lightweight multi-head attention, which processes the convolution feature map with attention mechanism. It is a downsampling operation, which is usually used to reduce the spatial resolution of the feature map while maintaining its important semantic information; At the same time, the SAM module receives these feature information and uses its zero-shot learning capability and the generated prompt module to segment the image by means of click prompts or object box prompts. The SAM model gradually optimizes the segmentation results according to the prompt information and handles the complex structures and objects in the image. Then, under the action of the few-shot fine-tuning module, the formula of the few-shot fine-tuning module is as follows:

[0017] in, represents the sigmoid function, represents element-level multiplication, GAP(·) and GMP(·) represent global average pooling and global maximum pooling operations in the spatial dimension, respectively. represents the feature representation after fine-tuning, Represents the input features before fine-tuning; The CNN and SAM models are fine-tuned using a small number of labeled samples. The fine-tuning process uses reinforcement learning methods to adjust the model parameters based on the model's initial segmentation results and doctors' feedback to improve its segmentation accuracy under few-sample conditions. In each training, the SAM model automatically generates the prompt module formula based on the fine-tuned feature information and prompt optimization mechanism as follows:

[0018] in, Represents the generated prompt information. Represents a fully connected layer for mapping operations, Represents the activation function used to map the input to a value between 0 and 1, which is used to control the output strength of the model. represents the query vector, represents the key vector, represents the value vector, Represents the transpose operation, which transposes the vector. Represents the dimension of the feature, used to scale the dot product result, is the frozen parameter, is a learnable parameter; Improve the segmentation effect and gradually improve the performance through iterative training. After multiple rounds of optimization and feedback, the model can eventually achieve high segmentation accuracy under the condition of few samples, which is suitable for medical image segmentation tasks in actual clinical diagnosis. Ultimately, after iterative learning and real-time feedback from doctors, the model can automatically update its weights and gradually improve the segmentation accuracy, outputting the final high-quality medical image segmentation results.

[0019] Example The data of the embodiment uses the BUSI breast segmentation dataset, the ACDC heart dataset, and the ISBI fundus image segmentation dataset, and uses the BUSI breast segmentation dataset, the ACDC heart dataset, and the ISBI fundus image segmentation dataset to preprocess and enhance the images to adapt to the deep learning training requirements; Data augmentation includes random flipping, rotation, translation, scaling, and affine transformation operations to improve the model's adaptability to different medical image variations. Basic cropping and scaling preprocessing are performed on the test images to ensure that the input images have a uniform size. The model is based on a combination of CNN and SAM frameworks, and introduces a few-sample fine-tuning module to avoid overfitting and improve training efficiency. During training, images were cropped to a uniform size of 256×256 to fit the model input. A strict training schedule was implemented, using the Adam optimizer with a learning rate set to 1e-5, a gamma value set to 0.99, and a batch size of 32 for training; The entire training process includes 100 epochs, each epoch samples 1500 steps, and hyperparameters are adjusted every 5 epochs. The model parameters are optimized through grid search to improve performance. In order to adapt to the few-sample segmentation task, a small amount of labeled data is used for fine-tuning, and the generation prompt module of SAM is combined with the CNN feature extraction module, so that the model can quickly learn on limited samples and achieve high-precision segmentation. The Dice coefficient is used as the main performance indicator for model evaluation to measure the consistency between the predicted and the true segmentation. Through this method, excellent segmentation performance is achieved on the BUSI breast segmentation dataset, the ACDC heart dataset, and the ISBI fundus image segmentation dataset, and the generalization ability and robustness of the model under few-sample conditions are verified.

[0020] like Figure 2 As shown, the few-sample medical image segmentation method combining CNN and SAM includes the following steps: Step 1: After extracting medical images from the database, first, preprocess the images to ensure that they meet the input specifications of the deep learning model. These preprocessing steps include cropping and normalization to adapt to the input requirements of the model; Step 2: The image is initially processed by the CNN feature extraction module. The CNN module encodes the input medical image and extracts the deep features of the image, paying special attention to the local information of key areas (such as tumors, organs, etc.). These features are then input into the image encoder in the SAM segmentation module. The image encoder maps the features of the image to the feature space in preparation for further segmentation processing; Step 3: The prompt generation module provides feedback to the SAM model through non-interactive prompts. The prompt encoder maps the input prompt information to the prompt feature space so that SAM can refine the image segmentation according to these prompts. By combining the image features extracted by CNN and the generated prompt information, SAM can quickly optimize under few-sample conditions and accurately segment key targets in medical images. Step 4: The module further improves the segmentation accuracy through the few-sample fine-tuning module, combining a small amount of labeled data and the prompt information output by the model. The formula of the few-sample fine-tuning module is as follows:

[0021] in, represents the sigmoid function, represents element-level multiplication, GAP(·) and GMP(·) represent global average pooling and global maximum pooling operations in the spatial dimension, respectively. represents the feature representation after fine-tuning, Represents the input features before fine-tuning; Step 5: The few-sample segmentation method of CNN and SAM can achieve efficient and accurate medical image segmentation under few-sample conditions through parallel processing, enhance the model's learning ability for small amounts of data, and improve the accuracy and robustness of the segmentation results.

[0022] As an implementation mode, the training unit trains the SAM model using the following formula:

[0023]

[0024]

[0025] Among them, y in CE is the true label, which takes a value of 0 or 1, p is the positive class probability of the model, which is usually the value output by sigmoid and ranges between [0,1]. A and B in Dice are two sets, usually the intersection of the predicted and true labels. is the intersection between the predicted and true labels, and are the sizes of set A and set B respectively; In MSE, N is the number of samples, is the true value, is the predicted value.

[0026] By generating hints and a few-shot fine-tuning mechanism, the accuracy and efficiency of medical image segmentation are greatly improved without the need for a large amount of labeled data, and high performance can be maintained, especially when data is scarce. By combining the powerful feature extraction capability of CNN with the zero-shot learning and hint generation mechanism of SAM, high efficiency and adaptability in medical image segmentation tasks are achieved. The hint generation module dynamically generates optimization hints based on the input image features and real-time feedback, and guides SAM to fine-tune the segmentation task under few-shot conditions. This iterative learning process enables the model to gradually improve the accuracy and robustness of the segmentation results by continuously updating and adjusting the segmentation strategy with the support of limited labeled data. The few-shot fine-tuning module optimizes CNN and SAM to improve the model's adaptability to a small amount of labeled data, ensuring efficient execution of medical image segmentation tasks in few-shot scenarios.

[0027] Compared with the existing technologies, the uniqueness of this method lies in combining the powerful feature extraction ability of CNN and the generation prompt mechanism of SAM, especially achieving efficient and accurate medical image segmentation under few-sample conditions, through parallel processing and few-sample fine-tuning; With the support of limited annotated data, this method can quickly learn and adapt to different medical image segmentation tasks. The dynamically generated prompt information and automatically optimized segmentation strategy enable the model to achieve high segmentation accuracy in a small number of iterations, and can cope with different image modalities and changing data sets while maintaining consistent segmentation effects. This flexible learning and adjustment process not only improves the segmentation accuracy, but also significantly enhances the generalization ability and reliability of the model on different data sets.

[0028] This method realizes a few-sample medical image segmentation method combining CNN and SAM. Through the method of the present invention, high-precision segmentation of medical images can be achieved with the support of a small amount of labeled data, and the method can effectively cope with medical imaging data of different modalities and large variability. By parallel processing CNN feature extraction and SAM generation prompt mechanism, the segmentation strategy can be dynamically optimized under a limited number of samples, and the segmentation accuracy and robustness can be significantly improved. This method not only maintains a high segmentation effect under a few-sample condition, but also can flexibly adapt to different medical image tasks, and enhances the generalization ability and reliability of the segmentation model. Therefore, the few-sample medical image segmentation method combining CNN and SAM of the present invention is efficient, accurate and widely applicable.

[0029] The experiment can be applied to the segmentation method of data sets of different imaging modes. Table 1 compares SAM and this embodiment, as follows: Table 1

[0030] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A few-sample medical image segmentation method combining CNN and SAM, characterized in that: include: Construction steps: Construct a SAM model for few-sample medical images combining CNN and SAM. The model includes a CNN feature extraction module, a few-sample fine-tuning module, and a prompt generation module. The CNN feature extraction module uses a convolutional neural network to extract features from preprocessed medical images. CNN filters complex information through multi-scale linear attention. Through multi-layer convolution and pooling operations, it automatically learns deep spatial features from medical images, especially the extraction of complex structures. In the few-sample learning scenario, the few-sample fine-tuning module maps the learned features to different feature spaces and then weights them through a gate mechanism. A small number of labeled samples are used for fine-tuning. By utilizing the powerful transfer learning ability of SAM, a small amount of data is optimized based on the existing pre-trained model, so that the model can effectively learn and segment when labeled samples are scarce. Training step: using cross entropy loss, Dice loss and MSE loss to train the SAM model to obtain a trained SAM model, wherein the parameters of the image encoder are frozen during the training process to improve training efficiency and reduce computing resource requirements; Segmentation steps: The medical image is first processed preliminarily, and the processed images are input into the CNN feature extraction module and the SAM segmentation module respectively. The CNN module generates and extracts local features in the medical image through the convolution layer, focusing on the detailed information of the key areas in the image. At the same time, the SAM segmentation module generates a prompt module through it, using a small number of labeled samples to guide the model to perform accurate target area segmentation by clicking prompts. SAM optimizes the segmentation results according to the input prompt information, and combines it with the features extracted by CNN, and finally outputs accurate medical image segmentation results.

2. The method for few-sample medical image segmentation combining CNN and SAM according to claim 1, characterized in that: The CNN feature extraction module, the few-sample fine-tuning module and the prompt generation module, The extraction formula of the CNN feature extraction module is: ； in, is a convolution operation, applied On the input, it is used to extract local information in images and feature maps. It represents a lightweight multi-head attention, which processes the convolution feature map with attention mechanism. It is a downsampling operation, which is usually used to reduce the spatial resolution of the feature map while maintaining its important semantic information; The formula for the few-shot fine-tuning module is as follows: ； in, represents the sigmoid function, represents element-level multiplication, GAP(·) and GMP(·) represent global average pooling and global maximum pooling operations in the spatial dimension, respectively. represents the feature representation after fine-tuning, Represents the input features before fine-tuning; The formula for generating the prompt module is as follows: ； in, Represents the generated prompt information. Represents a fully connected layer for mapping operations, Represents the activation function used to map the input to a value between 0 and 1, which is used to control the output strength of the model. represents the query vector, represents the key vector, represents the value vector, Represents the transpose operation, which transposes the vector. Represents the dimension of the feature, used to scale the dot product result, is the frozen parameter, is a learnable parameter.

Images