Polyp image segmentation method and system based on combination of SAM model and graph neural network

Abstract

The application discloses a polyp image segmentation method based on combination of a SAM model and a graph neural network, and comprises the following steps: acquiring an image to be segmented; inputting the acquired image to be segmented into a trained image segmentation model to output an image segmentation result; wherein the training of the image segmentation model comprises the following steps: constructing a training data set; constructing an image graph convolution attention encoder module to perform local graph structure feature extraction; based on a pre-trained SAM model, introducing a space and channel double adapter module to fine-tune the image encoder of the SAM model, and extracting global graph structure features by using the fine-tuned SAM model; fusing the extracted local graph structure features and global graph structure features; generating a segmentation result based on the fused image feature representation; and training by using the training data set to output a trained image segmentation model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image processing, and particularly relates to a technology for precise automatic segmentation of polyps in the diagnosis of colorectal cancer (CRC), specifically a polyp image segmentation method and system based on the combination of a SAM model and a graph neural network. BACKGROUND

[0002] Colorectal cancer is the second leading cause of cancer death worldwide, and is usually manifested as polyps in the early stage. In the field of medical image processing, especially in the early diagnosis of colorectal cancer, polyp segmentation technology is one of the key technologies. Therefore, early detection and accurate segmentation of polyps play a crucial role in improving the survival rate of patients.

[0003] Currently, polyp segmentation technology mainly relies on deep learning models, especially models based on U-Net and its variants. The U-Net model is a popular convolutional neural network (CNN) architecture that has been widely used due to its excellent performance in image segmentation tasks. These models effectively identify and segment polyps and other lesion regions by capturing local features and contextual information in images. In recent years, models based on Transformers have also been introduced into the polyp segmentation task due to their ability to handle long-range dependencies.

[0004] However, existing deep learning models still face several challenges in dealing with early polyps, especially in the segmentation of polyps with ambiguous boundaries, complex structures, or small sizes. These challenges mainly include:

[0005] 1. Ignoring structural connections and pixel region interactions: Existing models, especially those based on U-Net, tend to focus on capturing local features and boundary information, while ignoring the complex structural connections between polyps and surrounding tissues and the interactions between pixel regions, which may lead to the model's inability to accurately identify and segment polyps with unclear boundaries.

[0006] 2. Insufficient detection of small and early polyps: Early polyps are usually small in size and have little difference from the surrounding tissues, making it difficult for existing models to accurately detect and segment these polyps, increasing the risk of missed or misdiagnosed.

[0007] 3. Sensitivity to complex backgrounds: The complexity of the background in polyp images (such as bleeding, inflammation, etc.) may interfere with the performance of the model, leading to inaccurate segmentation results.

[0008] 4. Demand for computing resources: Although models based on Transformers can handle long-range dependencies, these models usually require high computing resources, which may not be practical in actual clinical applications. SUMMARY

[0009] In order to solve the limitations of existing medical image processing techniques in accurately segmenting colorectal polyps, especially the challenges in dealing with early-stage, small-size or complex-structure polyps, the present application provides a polyp image segmentation method and system based on the combination of SAM model and graph neural network, which significantly improves the detection accuracy, robustness and computational efficiency of the model in early diagnosis of colorectal cancer by using a graph attention mechanism and a SAM (Segment Anything Model) hybrid network architecture.

[0010] According to an aspect of the present application, a polyp image segmentation method based on the combination of SAM model and graph neural network is provided, comprising:

[0011] Obtaining an image to be segmented;

[0012] Inputting the obtained image to be segmented into a trained image segmentation model to output an image segmentation result; wherein the training of the image segmentation model comprises:

[0013] Constructing a training data set;

[0014] Constructing an image graph convolution attention encoder module for local graph structure feature extraction;

[0015] Based on the pre-trained SAM model, a spatial and channel dual adapter module is introduced to fine-tune the image encoder of the SAM model, and the fine-tuned SAM model is used to extract global graph structure features;

[0016] Fusing the extracted local graph structure features and global graph structure features;

[0017] Generating a segmentation result based on the fused image feature representation;

[0018] Using the training data set for training to output a trained image segmentation model.

[0019] As a further technical solution, the construction of the training data set comprises:

[0020] Collecting a diversified polyp image data set, including pathological images and related annotation masks;

[0021] Preprocessing based on the collected images;

[0022] Converting the preprocessed images into graph data.

[0023] As a further technical solution, the constructed image graph convolution attention encoder module comprises a four-layer graph convolution network and a one-layer graph attention network, wherein the graph convolution network is used for feature extraction, and the graph attention network is used to strengthen the feature representation between nodes.

[0024] As a further technical solution, a space and channel dual adapter module is introduced to fine-tune the image encoder of the SAM model, including:

[0025] The space and channel dual adapter module is inserted into the Transformer module of the SAM model, the channel adapter reduces the spatial dimension through average pooling and converts the features through a multilayer perceptron; the space adapter adjusts the dimension of the input features through convolution and deconvolution;

[0026] The channel adaptation result and the space adaptation result are fused to adapt the fine-tuned SAM model to the polyp image.

[0027] As a further technical solution, based on the fused image feature representation, a segmentation result is generated, including:

[0028] The prompt encoder is used to guide the model to focus on the key areas in the image according to the predefined prompt;

[0029] The mask decoder of the SAM model receives the image feature representation and converts it into a segmentation mask with the same resolution as the input image.

[0030] According to an aspect of the present application, a polyp image segmentation system based on the combination of a SAM model and a graph neural network is provided, including:

[0031] An image input module is used to obtain an image to be segmented;

[0032] An image segmentation module is used to input the obtained image to be segmented into a trained image segmentation model, and output an image segmentation result; wherein the training of the image segmentation model includes:

[0033] A training data set is constructed;

[0034] An image graph convolution attention encoder module is constructed to extract local graph structure features;

[0035] Based on the pre-trained SAM model, a space and channel dual adapter module is introduced to fine-tune the image encoder of the SAM model, and the fine-tuned SAM model is used to extract global graph structure features;

[0036] The extracted local graph structure features and global graph structure features are fused;

[0037] Based on the fused image feature representation, a segmentation result is generated;

[0038] The training data set is used for training, and a trained image segmentation model is output.

[0039] According to an aspect of the present specification, a polyp image segmentation device based on a combination of a SAM model and a graph neural network is provided, comprising a memory and a processor, the memory storing program instructions executed by the processor, and the processor calling the program instructions to execute the steps of the polyp image segmentation method based on the combination of the SAM model and the graph neural network.

[0040] According to an aspect of the present specification, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions causing the computer to execute the steps of the polyp image segmentation method based on the combination of the SAM model and the graph neural network.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] The image segmentation model (Polyp-GSAM) of the present application significantly improves the accuracy and efficiency of polyp segmentation in medical images through its innovative graph attention guided hybrid SAM architecture. This model skillfully integrates local structural features and global information, significantly enhancing the detection ability of early and complex structured polyps, effectively reducing the occurrence of misdiagnosis and missed diagnosis. At the same time, this model optimizes the use of computing resources and improves processing speed, making it more suitable for real-time clinical application scenarios.

[0043] Specifically, the Polyp-GSAM model also includes a fine-tuned CS-Adapter module that enhances the model's adaptability to different medical image characteristics, further improving the model's generalization ability. In addition, its optimized feature fusion strategy effectively reduces the risk of overfitting, ensuring the model's consistent performance on diverse data sets.

[0044] Overall, the present application not only significantly improves the quality and efficiency of medical diagnosis, but also provides clinicians with a more reliable and convenient diagnostic tool, greatly promoting the development of early colorectal cancer diagnosis technology. Through these technological innovations, the present application not only improves the effectiveness of medical services, but also helps to save medical resources, which is of great significance to improving public health. BRIEF DESCRIPTION OF DRAWINGS

[0045] To make the technical solutions in the embodiments or prior art of the present application clearer, the drawings used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0046] Figure 1A flowchart of an image segmentation process provided by an embodiment of the present application.

[0047] Figure 2 An architecture diagram of an image segmentation model provided by an embodiment of the present application.

[0048] Figure 3 A diagram of an image graph convolution attention encoding module provided by an embodiment of the present application.

[0049] Figure 4 A diagram of a spatial adapter module and a channel adapter module provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] The method provided by the present application is specifically applied to the field of endoscopy and medical imaging, and automatically identifies and accurately segments colorectal polyps through advanced image analysis methods. The application of this technology not only improves the accuracy of polyp detection, but also greatly improves the efficiency of diagnosis and treatment, thereby helping to improve the survival rate and quality of life of patients.

[0051] The present application proposes an innovative image segmentation method in view of the limitations of existing medical image processing techniques in accurately segmenting colorectal polyps, especially the challenges in processing early-stage, small-size or complex-structure polyps. The method uses a graph attention mechanism and a SAM hybrid network architecture to significantly improve the detection accuracy, robustness and computational efficiency of the model in early diagnosis of colorectal cancer.

[0052] The above technical innovations not only solve the accuracy and robustness problems of polyp segmentation in the prior art, but also improve the practicality and efficiency of operation, making the present application have important application value in early diagnosis of colorectal cancer.

[0053] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other to form new technical solutions, and such combination is not subject to the order of steps and / or structure composition mode, but must be based on the realization by those skilled in the art. When the combination of technical solutions contradicts each other or cannot be realized, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0054] The embodiment of the present application provides a polyp image segmentation method based on the combination of a SAM model and a graph neural network, first, an image to be segmented is acquired; then, the acquired image to be segmented is input into a trained image segmentation model (i.e., a Polyp-GSAM model), and an image segmentation result is output.

[0055] The Polyp-GSAM model provided by the embodiment of the present application is a graph attention guided hybrid SAM (Segment Anything Model) architecture model, which aims to realize accurate segmentation of polyps in medical images. The model comprehensively uses deep learning, graph theory and attention mechanism to improve the detection and segmentation accuracy of polyps, especially when dealing with early, small size or complex structure polyps. The model mainly includes an image graph convolution attention encoding module, a spatial and channel dual adapter module, an image encoder module, a feature fusion module, a mask decoder module and a prompt word encoder module.

[0056] In the field of image segmentation, especially medical image segmentation such as polyp detection and segmentation, although traditional convolutional neural networks (CNN) and attention mechanisms have achieved certain results, it is often difficult to capture complex structural information and pixel-to-pixel relationships in images. In order to improve the accuracy and robustness of segmentation, the embodiment of the present application proposes a novel image graph convolution attention encoder module.

[0057] Specifically, the image graph convolution attention encoding (IGCA) module is responsible for converting the input medical image into graph structure data. In this process, each pixel point is regarded as a node in the graph, and the connection relationship between nodes is established based on the neighborhood relationship between pixels. The image graph convolution attention encoding (IGCA) module uses graph convolution network (GCN) and graph attention network (GAT) to extract local features and deep structure features of the image, so as to capture the complex interaction relationship between pixels.

[0058] The goal of the IGCA module is to convert the graph structure data into high-dimensional feature representation, focusing on local image features. The module is composed of four layers of graph convolution network (Graph Convolutional Network) and graph attention network (Graph Attention Network).

[0059] The graph convolutional network (GCN) captures local features by simulating the adjacency relationship between pixels in the image. The features of each node (pixel) not only contain its own information, but also contain the information of its neighbor nodes, which helps the model understand the spatial structure in the image. In GCN, the weights of the convolution kernel are shared across all nodes and edges, which reduces the number of parameters of the model and makes the model more efficient. GCN can be stacked with multiple layers, each of which can capture features of different scales, thereby realizing the fusion of multi-scale features. The working principle of GCN is based on the aggregation operation, in which the new features of each node are updated by aggregating the features of itself and its neighbor nodes.

[0060] After passing through the GCN layer, the final feature representation will be input to the GAT layer to further strengthen the feature representation between nodes. GAT assigns different weights to different nodes through the attention mechanism, which enables the model to pay more attention to important features in the image. GAT does not rely on the fixed order or distance metric of nodes, so it is more flexible and can capture more complex dependencies. The calculation of GAT can be parallelized, which makes it more efficient on large-scale graphs. GAT updates the features by calculating the attention coefficients of each node to other nodes.

[0061] By combining GCN and GAT, the image graph convolutional attention encoder module can effectively extract local features and global structure information of the image, providing a powerful feature representation for the polyp segmentation task. This combination takes advantage of the local connectivity of GCN and the dynamic attention mechanism of GAT, enabling the model to more accurately identify and segment polyp regions.

[0062] With the development of deep learning technology, significant progress has been made in the field of image segmentation. The SAM model is an advanced image segmentation model that can handle various image segmentation tasks. However, medical images differ significantly from natural images in terms of color, texture, and pixel intensity, and direct application of the SAM model to medical image segmentation does not yield satisfactory results. Therefore, it is necessary to fine-tune the SAM model to adapt to the characteristics of medical images.

[0063] In order to introduce the relevant knowledge of polyp images into the SAM model and make the SAM encoder better adapt to the characteristics of polyp images, the spatial and channel dual adapter module is introduced to fine-tune the image encoder of the SAM, so that the SAM can learn new knowledge based on the original knowledge. The spatial and channel dual adapter module is inserted into the Transformer Blocks of the SAM encoder, which adjusts the channel and spatial dimensions of the feature map respectively. This fine-tuning process enables the model to more accurately capture the features of polyps, improving the accuracy of segmentation.

[0064] The feature fusion module integrates the local graph structure features extracted by the IGCA module with the global features extracted by the SAM encoder at the feature fusion layer to obtain a more comprehensive image feature representation. This fusion strategy not only enhances the model's understanding of complex polyp structures but also optimizes the integration of features, thereby improving segmentation performance and generalization ability.

[0065] The prompt encoder module is used to process additional information related to the segmentation task, such as bounding boxes or click positions, to further improve segmentation accuracy. This module enhances the model's response to user input, making it perform better in interactive segmentation tasks.

[0066] The mask decoder module is used to further process the fused features to generate the polyp segmentation mask. The mask decoder is responsible for converting the integrated features into the final segmentation result, thereby achieving accurate positioning and segmentation of polyps.

[0067] The polyp image segmentation method based on the combination of the SAM model and the graph neural network provided by the embodiment of the present application introduces the graph neural network into the SAM model to solve the problem of insufficient understanding of complex medical images by the SAM. The zero-shot segmentation capability of the SAM can greatly improve the segmentation efficiency of the model, reduce the consumption of computing resources, and provide a referenceable segmentation mask for professional doctors to assist them in treatment.

[0068] Please refer to Figure 1 and Figure 2 which give the flowchart and model architecture diagram of the graph attention guided hybrid SAM architecture model. The embodiment of the present application adopts a dual network encoder structure SAM image encoder (SAM Image Encoder) and image graph convolution attention encoder module (IGCA). After the image is extracted by the double-layer encoder, it enters the mask decoder (Mask Decoder), and then the prompt word encoder (Prompt Encoder) gives the point and box prompt to guide the mask decoder to give the segmented mask. The training process of the model includes the following steps:

[0069] Step 1: Preparation and preprocessing of the data set.

[0070] Step 1.1, construction of the data set: collect a diversified polyp data set, including pathological images and related annotation masks, to ensure the robustness of the SAM model. The data set is composed of multiple public polyp image data sets, including ETIS-Larib, CVC-ClinicDB, CVC-ColonDB, CVC-300, and Kvasir-SEG. These data sets provide different numbers of polyp images and their corresponding pixel-level annotations, and the resolutions are also different.

[0071] Step 1.2, data set division: using the recognized data set division method, 900 Kvasir-SEG images and 550 CVC-ClinicDB images, a total of 1450 samples, are used as the training set; the test set includes the remaining 100 images of Kvasir-SEG, the remaining 62 images of CVC-ClinicDB, 380 samples of CVC-ColonDB, 196 samples of ETIS, and 60 images of CVC-300.

[0072] Step 1.3, data set preprocessing, including cropping, rotating, deforming and other image enhancement operations.

[0073] Further comprising:

[0074] Step 1.3.1, image size adjustment, the input image size is uniformly adjusted to 256x256 pixels, which helps the input consistency of the model.

[0075] Step 1.3.2, using an image-to-graph data conversion module to convert the input image into graph structure data (i.e. the data of nodes and edges that the image graph convolution attention encoding (IGCA) module can identify), which involves converting each pixel point of the image into a node in the graph, and establishing edges according to the neighborhood relationship of the pixel points.

[0076] Step 1.3.3, by grayscale processing, the color image is converted into a grayscale image, so as to better capture the local features and structural information of the image.

[0077] For each image I, where I has a shape of [C, H, W], it is converted into a grayscale image G, which is calculated as follows:

[0078]

[0079] where I C represents the Cth channel of the image, C represents the number of channels, H and W represent the height and width of the image respectively. For each pixel p xy in the grayscale image G, a node v is created at position (x, y), and the feature f(v) of the node is the grayscale value of the pixel. For each pixel in the image, edges are created to connect it with its up, down, left and right neighbors. If the pixel p xy is located at position (x, y), for each neighbor that exists, an edge (v xy , v neighbor ) is added, where v neighbor is the node corresponding to the neighbor pixel, and the specific edge set can be represented as:

[0080] E={(v xy ,v x-1,y ),(vxy v x+1,y ),(v xy ,v x,y-1 ),(v xy ,v x,y+1 )}

[0081] Finally, for the whole image, a graph G = (V, E) is obtained, where V is the set of all nodes defined by the pixel gray value, and E is the set of all edges defined by the pixel adjacency relationship.

[0082] Step 2: The image graph convolution attention encoder module performs graph feature extraction.

[0083] Referring to Figure 3 , the image graph convolution attention encoder module is composed of 4 layers of graph neural network (GCN) and one layer of GAT (graph attention mechanism).

[0084] The working principle of GCN is based on aggregation operation, in which the new feature of each node is updated by aggregating the features of itself and its neighbor nodes. Specifically, for the graph convolution network part, there are L GCN layers, where L = num_gcn_layers. For each node v in the graph, its feature representation X can be updated as follows.

[0085]

[0086] where N(v) is the set of neighbor nodes of node v, c uv is the normalization constant, and W l is the weight matrix of the Lth layer, is the feature representation of node u in the (l-1)th layer, is the initial feature of node v (e.g., pixel intensity), and ReLU is an activation function used to introduce nonlinearity.

[0087] The graph attention network GAT assigns different weights to different nodes through the attention mechanism, which enables the model to pay more attention to important features in the image.

[0088] For each node v in the graph, its updated feature X IGCA after the GAT layer can be calculated as follows:

[0089]

[0090] where, where the attention coefficient of node u to node v, k is the number of network layers, W is the weight matrix, and σ is the softmax function used to normalize the attention weight. The attention coefficient The calculation is as follows:

[0091]

[0092] where a is a learnable weight vector used to calculate the relative importance between node pairs, || denotes the concatenation operation, and σ is the softmax function used to normalize the attention weights from all neighbors of v. Each node undergoes a series of graph convolution and graph attention transformations, resulting in the final feature X IGCA , which contains comprehensive information about local neighborhoods and global structures. This allows the model to capture complex spatial relationships in image data, providing a powerful feature representation for downstream tasks.

[0093] By combining GCN and GAT, the image graph convolution attention encoder module can effectively extract local features and global structure information of images, providing a powerful feature representation for polyp segmentation tasks. This combination takes advantage of the local connectivity of GCN and the dynamic attention mechanism of GAT, allowing the model to more accurately identify and segment polyp regions.

[0094] Step 3: Fine-tuning based on the SAM model.

[0095] With the development of deep learning technology, significant progress has been made in the field of image segmentation. The SAM model, as an advanced image segmentation model, can handle various image segmentation tasks. However, medical images differ significantly from natural images in terms of color, texture, and pixel intensity, and direct application of the SAM model to medical image segmentation does not yield satisfactory results. Therefore, it is necessary to fine-tune the SAM model to adapt to the characteristics of medical images.

[0096] The advantages of fine-tuning the SAM model include: improved adaptability, fine-tuning the SAM model allows it to better adapt to the characteristics of medical images, improving segmentation accuracy; reduced computational resources, by fine-tuning rather than completely retraining, reducing the consumption of computational resources; fast deployment, the fine-tuning process can quickly adapt to new datasets, speeding up the deployment and application of the model.

[0097] Please refer to Figure 4 The working principle of fine-tuning the SAM model is as follows: by using channel adapters (ChannelAdapter) and spatial adapters (Space Adapter), which are integrated into the Transformer blocks of the SAM model, to enhance the model's learning ability for medical image features without significantly changing the existing model architecture or parameters.

[0098] The detailed steps include:

[0099] Step 3.1, Pre-trained Model Selection: Select a SAM model that is pre-trained on a large-scale general image dataset to obtain a basic feature representation.

[0100] Step 3.2, Data Preparation: Collect and label a medical image dataset for the fine-tuning process.

[0101] Step 3.3, Freeze Parameters: Freeze most parameters in the SAM model, especially the weights at the bottom, to preserve the features learned by the model on general images.

[0102] Step 3.4, Introduce Spatial and Channel Adapters, only update these parameters during training.

[0103] The main purpose of the channel adapter is to adjust the channel dimension of the feature map to better adapt to the downstream task. This process is done through average pooling and a multi-layer perceptron (MLP).

[0104] Average Pooling: First, the input feature map X is reduced in spatial dimension through average pooling, which helps the model capture global information.

[0105] Feature Transformation: The pooled feature map is transformed through an MLP. The MLP contains two weight matrices W1 and W2, as well as ReLU activation functions and a Sigmoid function.

[0106] X channel = σ(W2(Relu(W1(AvgPool(X))))

[0107] where and are learned weight matrices, C' is the dimension of the intermediate layer, and σ is the Sigmoid function, which maps weights to between 0 and 1, indicating the importance of the channel.

[0108] The spatial adapter adjusts the spatial resolution of the input features through convolution and deconvolution operations.

[0109] X spatial = ConvTranspose2d(Relu(Conv2d(X channel )))

[0110] Conv2d and ConvTranspose2d are convolution and deconvolution operations used to adjust the spatial resolution of features.

[0111] Convolution: The output of the channel adapter is first passed through a convolution layer, Conv2d, to further extract spatial features.

[0112] Deconvolution: Then, the spatial resolution of the feature map is adjusted through the deconvolution layer ConvTranspose2d to match the spatial dimensions of the original input feature map.

[0113] Step 3.5, Feature Fusion: The feature fusion strategy fuses the outputs of the channel adapter and spatial adapter with the original input feature map to enhance the model's ability to integrate local and global information.

[0114] Element-wise multiplication: The output X of the channel adapter channel is element-wise multiplied with the original input feature map X to integrate channel attention information.

[0115] Weighted sum: The result of element-wise multiplication is weighted summed with the output X of the spatial adapter spatial to obtain the final fused feature map.

[0116] Specifically, the channel attention weight is applied to the original input, and the result of spatial attention transformation is fused, as shown in the formula:

[0117] X fusion = X channel ⊙ X + X spatial

[0118] If the skip connection is enabled, X spatial is added to the input X; otherwise, X spatial is directly used, where ⊙ represents element-wise multiplication.

[0119] Step 3.6, Loss function customization: Design a loss function suitable for medical image segmentation tasks, such as a combined loss function, to optimize the model's ability to recognize lesion areas.

[0120] Step 3.7, Data augmentation and cross-validation: Use data augmentation techniques to increase the diversity of medical image data and perform cross-validation to ensure that the model performs well on different datasets.

[0121] Step 4, Feature fusion.

[0122] Fuse the graph features obtained from the image graph convolution attention encoder module and the output features of the fine-tuned SAM graph encoder module, with the formula:

[0123] X fusion = W vit × X vit + W IGCA × X IGCA

[0124] W vit and WIGCA respectively, to maintain feature consistency, we let W vit and W IGCA add up to 1, X fusion denotes the result after fusion. Finally, a series of convolution and normalization operations are included in the Neck layer to output the final image feature representation.

[0125] Step 4, Mask Decoder and Prompt Encoder.

[0126] The mask decoder structure of SAM is retained, and the parameters are updated to process the final image feature representation to generate a predicted mask.

[0127] The mask decoder (MaskDecoder) includes:

[0128] Feature representation: The mask decoder receives high-dimensional feature representations from the image encoder and converts them into segmentation masks with the same resolution as the input image.

[0129] Convolution and normalization operations: A series of convolution layers and normalization layers are used to process features, enhance the expression ability of features, and gradually restore the spatial resolution of the image.

[0130] Up-sampling and fusion: Use up-sampling techniques (such as transposed convolution) to increase the spatial resolution of the feature map to the same as the input image, and use skip connections to fuse low-level features to retain more detailed information.

[0131] Prompt encoder (PromptEncoder):

[0132] Prompt enhancement: The prompt encoder uses predefined prompts (such as bounding boxes, key points, or natural language descriptions) to guide the model to focus on key areas in the image.

[0133] Feature fusion: Combine prompt information and image features to enhance the model's ability to identify key areas through attention mechanisms.

[0134] Multi-modal output: Provide multiple output modes, including BOX-based segmentation and Point-based segmentation, to adapt to different application scenarios and needs.

[0135] In practical applications, the mask decoder and the prompt encoder can be jointly trained to optimize the overall performance of the model. The mask decoder is responsible for generating accurate segmentation masks, while the prompt encoder provides additional context information to help the model better understand and segment target areas in the image.

[0136] Step 6, Model training.

[0137] Loss function definition: Define the loss function as a combination of weighted cross-entropy loss and Dice loss to optimize the segmentation performance of the model.

[0138] L = α·LWCE + β·LDice

[0139] Where α and β are the weight coefficients of the weighted cross-entropy loss and the Dice loss, respectively.

[0140] Optimizer selection: Choose the Adam optimizer for model training due to its adaptive learning rate characteristics suitable for processing complex medical image data.

[0141] Training period setting: Set the model training period to 50 epochs to ensure that the model learns sufficiently and avoids overfitting.

[0142] Learning rate adjustment: Set the initial learning rate to 1x10 -4 And use the learning rate decay strategy during training according to the validation set performance.

[0143] Batch size selection: Choose an appropriate batch size based on GPU memory capacity and dataset size to ensure the stability and efficiency of model training.

[0144] Data augmentation techniques: Apply rotation, flipping, scaling, and other data augmentation techniques to increase data diversity and improve the generalization ability of the model.

[0145] Cross-validation: Use cross-validation to evaluate model performance to ensure that the model performs well and stably on different datasets.

[0146] Hyperparameter definition: Design multiple hyperparameters such as fusion weights, Dropout, and GCN layer number, which need to be adjusted in real time according to specific experimental conditions.

[0147] Step 7, performance evaluation.

[0148] Use Dice Coefficient and Mean Intersection over Union as evaluation indicators. Run the model on a 4080 GPU with 16GB of memory.

[0149] Dice coefficient (Dice Coefficient) is also known as a variant of F1 score, which is a performance indicator for evaluating binary classification problems, especially in image segmentation tasks. It measures the similarity between the model's predicted segmentation area and the true labeled area. The value of the Dice coefficient ranges from 0 to 1. A value of 1 indicates perfect segmentation, i.e., the model's predicted segmentation area is completely consistent with the true labeled area; a value of 0 indicates no overlap and failed segmentation.

[0150] mIoU is the average of the intersection over union (IoU) for multiple classes, used for performance evaluation in multi-class image segmentation tasks. It measures the degree of overlap between the model's predicted segmentation regions and the true labeled regions. The value of mIoU also ranges from 0 to 1. A value of 1 indicates that all class segmentations are completely correct; a value of 0 indicates no overlap and a failed segmentation. mIoU considers the segmentation performance of all classes, so it is a global performance indicator.

[0151] The present invention successfully designs and implements a new hybrid architecture model named Polyp-GSAM, which is specifically designed for polyp segmentation tasks. This model is based on the core framework of SAM (Segment Anything Model) and integrates an innovative image graph convolution attention encoding module to facilitate collaborative training. By fine-tuning SAM for polyp images and allowing the image graph convolution attention encoding module to focus on extracting local structural features and integrating non-local pixel relationships, the method described in the present invention effectively combines these two feature types. This method creates a comprehensive feature representation that includes long-range dependencies and short-range structural information, significantly improving the model's performance on multiple standard polyp datasets. This success not only confirms the effectiveness of the collaborative operation between SAM and the image graph convolution attention encoding, but also indicates the potential for expansion of the method in the broader challenges of medical image segmentation.

[0152] The experimental results show that the Polyp-GSAM model outperforms existing state-of-the-art methods on multiple benchmark datasets, marking it as a powerful tool for early colorectal cancer intervention. In addition, the model described in the present invention also achieves good performance on the CVC-ColonDB, CVC-300, and ETIS-Larib datasets, which are not included in the training data, further demonstrating the model's robustness and generalization ability. Through quantitative and qualitative evaluation, the model described in the present invention performs well in terms of intersection over union and Dice coefficient, among other indicators. These findings collectively verify the effectiveness of our model in diverse data configurations.

[0153] In summary, the Polyp-GSAM model provides a new solution for the field of medical image segmentation, particularly in handling complex structures and subtle features, demonstrating superior performance.

[0154] The implementation basis of each embodiment of the present application is achieved by the programmed processing of a device with processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present application are packaged into various modules. Based on this actual situation, on the basis of the above-mentioned embodiments, the embodiment of the present application provides a polyp image segmentation system based on the combination of a SAM model and a graph neural network, which is used to execute the polyp image segmentation method based on the combination of a SAM model and a graph neural network in the above-mentioned method embodiment.

[0155] The system comprises: an image input module, configured to acquire a to-be-segmented image; an image segmentation module, configured to input the acquired to-be-segmented image into a trained image segmentation model and output an image segmentation result; wherein the training of the image segmentation model comprises: constructing a training data set; constructing an image graph convolution attention encoder module to perform local graph structure feature extraction; introducing a space and channel dual adapter module to fine-tune the image encoder of the SAM model based on a pre-trained SAM model, and extracting global graph structure features by using the fine-tuned SAM model; fusing the extracted local graph structure features and global graph structure features; generating a segmentation result based on the fused image feature representation; and training by using the training data set to output a trained image segmentation model.

[0156] The polyp image segmentation system based on the combination of a SAM model and a graph neural network provided by the embodiment of the present application is aimed at the limitations of existing medical image processing technology in accurate segmentation of colorectal polyps, especially the challenges in processing early-stage, small-size or complex-structure polyps. By using the above-mentioned several modules, the detection accuracy, robustness and computational efficiency of the model in early diagnosis of colorectal cancer are significantly improved by using the graph attention mechanism and the SAM hybrid network architecture.

[0157] It should be noted that the system embodiments provided by the present application are used to implement the methods in the above-mentioned method embodiments, and are also used to implement the methods in other method embodiments provided by the present application. The difference is only that the corresponding functional modules are set, and the principle is basically the same as that of the above-mentioned system embodiments provided by the present application. As long as the person skilled in the art, on the basis of the above-mentioned system embodiments, refers to the specific technical solutions in other method embodiments, obtains the corresponding technical means by combining technical features, and the technical solutions composed of these technical means, on the premise of ensuring the practicability of the technical solutions, improves the modules in the above-mentioned system embodiments to obtain the corresponding system class embodiments, which are used to implement the methods in other method class embodiments.

[0158] It should be understood that the implementation of the system embodiments of the present application can be implemented by referring to the above-mentioned method embodiments, which will not be repeated here.

[0159] Based on the same inventive concept as the foregoing embodiments, the present embodiment also provides a polyp image segmentation device based on the combination of a SAM model and a graph neural network, comprising a memory and a processor, the memory storing program instructions executed by the processor, and the processor calling the program instructions to execute the steps of the polyp image segmentation method based on the combination of the SAM model and the graph neural network.

[0160] Based on the same inventive concept as the foregoing embodiments, the present embodiment also provides a non-transitory computer-readable storage medium storing computer instructions, which cause the computer to execute the steps of the polyp image segmentation method based on the combination of the SAM model and the graph neural network.

[0161] In summary of the foregoing embodiments, the present application solves the problems of the prior art through the following technical innovations:

[0162] 1. Introducing an image graph convolutional encoder module: through the graph attention mechanism, the present application can more effectively handle long-distance dependencies and complex structures in images, thereby improving the recognition and segmentation ability of polyps with fuzzy boundaries or complex structures.

[0163] 2. Using a SAM hybrid network architecture: the SAM architecture combines the multi-scale processing capability of deep learning and the detailed analysis of specific regions, enhancing the model's detection ability for small-sized polyps and improving the segmentation accuracy in complex backgrounds.

[0164] 3. Introducing a spatial and channel dual adapter module to fine-tune the SAM, which uses a small amount of computing resources to complete the learning of polyp image features by SAM.

[0165] 4. Optimizing computing efficiency: this architecture design also considers the computing resource limitations in actual clinical applications, reducing the dependence on high-performance computing resources by optimizing algorithms and network structures, making the present application more suitable for deployment in resource-constrained environments.

[0166] The above technical innovations not only solve the accuracy and robustness problems of polyp segmentation in the prior art, but also improve the practicality and efficiency of the operation, making the present application have important application value in the early diagnosis of colorectal cancer.

[0167] The terms "comprise" and "have" and any variations thereof in the specification and claims of the present application and the above-described drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to the steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0168] It should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A polyp image segmentation method based on a combination of a SAM model and a graph neural network, characterized in that, The method comprises the following steps: acquiring an image to be segmented; inputting the acquired image to be segmented into a trained image segmentation model to output an image segmentation result; wherein the training of the image segmentation model comprises the following steps: constructing a training data set; constructing an image graph convolution attention encoder module to extract local graph structure features; the image graph convolution attention encoder module uses a graph convolution network and a graph attention network to extract local features and deep structure features of an image, so as to capture complex interaction relationships between pixels; the graph convolution network captures local features by simulating the adjacency relationship between pixel points in the image; the feature representation after the graph convolution network is input into the graph attention network; the graph attention network assigns different weights to different nodes through an attention mechanism, and updates the features by calculating the attention coefficients of each node to other nodes; based on a pre-trained SAM model, a space and channel dual adapter module is introduced to fine-tune the image encoder of the SAM model, and a global graph structure feature is extracted by using the fine-tuned SAM model; the space and channel dual adapter module is introduced to fine-tune the image encoder of the SAM model, which comprises the following steps: inserting the space and channel dual adapter module into the Transformer module of the SAM model; the channel adapter reduces the spatial dimension through average pooling and converts the features through a multilayer perceptron; the space adapter adjusts the dimension of the input features through convolution and deconvolution; the channel adaptation result and the space adaptation result are fused to adapt the fine-tuned SAM model to the polyp image; fuse the extracted local graph structure features and global graph structure features to obtain image feature representation; generate a segmentation result based on the fused image feature representation; train the training data set to output a trained image segmentation model.

2. The polyp image segmentation method based on the combination of the SAM model and the graph neural network according to claim 1, characterized in that, The training data set is constructed, which comprises the following steps: collecting a diversified polyp image data set, including pathological images and related annotation masks; preprocessing the collected images; converting the preprocessed images into graph data.

3. The polyp image segmentation method based on the combination of the SAM model and the graph neural network according to claim 1, characterized in that, The constructed image graph convolution attention encoder module comprises four layers of graph convolution network and one layer of graph attention network; the graph convolution network is used for feature extraction, and the graph attention network is used for strengthening the feature representation between nodes.

4. The polyp image segmentation method based on the combination of the SAM model and the graph neural network according to claim 1, characterized in that, Based on the fused image feature representation, a segmentation result is generated, which comprises the following steps: using a prompt encoder to guide the model to focus on the key areas in the image according to a pre-defined prompt; using a mask decoder of the SAM model to receive the image feature representation and convert it into a segmentation mask with the same resolution as the input image.

5. A polyp image segmentation system based on the combination of a SAM model and a graph neural network, for implementing the method of any one of claims 1 to 4, characterized in that, The method comprises the following steps: an image input module is used to acquire an image to be segmented; an image segmentation module is used to input the acquired image to be segmented into a trained image segmentation model to output an image segmentation result; wherein the training of the image segmentation model comprises the following steps: constructing a training data set; constructing an image graph convolution attention encoder module to extract local graph structure features; based on a pre-trained SAM model, a space and channel dual adapter module is introduced to fine-tune the image encoder of the SAM model, and a global graph structure feature is extracted by using the fine-tuned SAM model; The extracted local graph structure features are fused with global graph structure features to obtain an image feature representation; Based on the fused image feature representation, a segmentation result is generated; The training data set is used for training, and a trained image segmentation model is output.

6. A polyp image segmentation device based on a combination of a SAM model and a graph neural network, characterized by, The memory stores program instructions executed by the processor, and the processor invokes the program instructions to perform the steps of the polyp image segmentation method based on the SAM model combined with the graph neural network according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, which cause the computer to perform the steps of the polyp image segmentation method based on the SAM model combined with the graph neural network according to any one of claims 1 to 4.

Images