Colonoscopy polyp image detection method based on Mamba and YOLOv8
By combining Mamba and YOLOv8 models, the network architecture is improved, and the problems of high missed detection rate and unstable accuracy in colonoscopic polyp detection are solved, and more efficient polyp feature extraction and detection are achieved, which is suitable for clinical applications.
Patent Information
- Application Number
- CN202411244197.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-09-06
AI Technical Summary
The prior art has problems of high missed detection rate and unstable detection accuracy in colonoscopic polyp detection, especially due to the recognition bias caused by complex polyp morphology and differences in doctor experience, and the traditional target detection model has weak learning ability of polyp image features.
Combining Mamba and YOLOv8 models, using Mamba's long-range dependency capture capability and convolutional local feature extraction capability, improve the YOLOv8 network architecture, add Mamba attention mechanism, enhance feature representation ability, reduce computational complexity, and improve the accuracy and efficiency of polyp detection .
Effectively capture the coarse and fine-grained characteristics of polyps, improve the accuracy of colonoscopic polyps image detection, reduce the computational complexity and parameter quantity, and is suitable for clinical application needs.
Smart Images

Figure CN118762009B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image detection, and in particular relates to a colonoscopy polyp image detection method based on Mamba and YOLOv8. Background Art
[0002] Colorectal cancer (CRC) is one of the most deadly and concerning cancers. According to published literature, CRC is now the third most common cancer in the world and the fourth most deadly cancer. The survival rate of CRC patients depends on the stage at which it is detected, ranging from a survival rate of more than 95% in stage I to less than 35% in stages IV and V; therefore, it is important to detect it at an early stage using screening techniques such as colonoscopy.
[0003] Colonoscopy is one of the most effective and direct methods for detecting colorectal cancer. Using colonoscopy images, doctors can directly observe the pathological characteristics of polyps. Early detection of polyps can prevent colorectal cancer early. However, in clinical practice, the missed detection rate of polyps during colonoscopy is still as high as 6%. Among them, there are two key factors that affect the diagnosis results: 1) Objective factors. The morphology of polyps is relatively complex, with large and small shapes, and small polyps that are not easily noticed are difficult to detect; 2) Subjective factors. Differences in doctor's experience and personal status will lead to deviations in polyp detection results. Therefore, there is instability in manual detection methods, and polyp detection and identification vary greatly among individual doctors, and the detection accuracy still needs to be improved.
[0004] At present, there are few polyp detection methods based on deep learning. Most of them directly apply traditional target detection models in clinical practice, which lacks task adaptability. As one of the mainstream methods of target detection, the YOLO algorithm has the characteristics of high computing speed, low computing complexity, and high precision. However, due to the data diversity and camouflage of polyp images, the effect of applying the YOLO algorithm to polyp images is poor, and the feature learning ability of polyp images is weak. As a selective state space model, Mamba has attracted much attention for its ability to model long-distance dependencies under linear complexity. Mamba was originally designed to handle sequence-to-sequence learning tasks, but its application potential in the field of computer vision has gradually been discovered. Mamba is combined with traditional target detection models to improve the accuracy and efficiency of polyp detection. Although there is no mature solution in this field, this research direction undoubtedly provides new ideas and possibilities for future polyp detection tasks. Summary of the invention
[0005] In order to solve the above technical problems, the present invention provides a high-precision colonoscopic polyp detection method based on Mamba and YOLOv8, which makes full use of the long-range dependency capture ability of the Mamba model and the local feature extraction ability of convolution, and combines the YOLOv8 network architecture to make the learned feature representation powerful and generalizable, greatly reduce the computational complexity and have high efficiency, thereby improving the accuracy of colonoscopic polyp image detection.
[0006] The technical solution of the present invention to solve the above technical problem is: a colonoscopic polyp image detection method based on Mamba and YOLOv8, comprising the following steps:
[0007] S1, obtain colonoscopic polyp image data and randomly divide it into training validation set and test set;
[0008] S2, performs data preprocessing operations on the sample set divided by S1, including size standardization, rotation, hue change and noise addition operations;
[0009] S3, using the pre-processed training sample set and verification sample set obtained in step S2 to train and verify the neural network model, and save the trained neural network model; specifically as follows: in the Mamba and YOLOv8 neural network model encoding part, the sample sets obtained in steps S1 and S2 are input into the Mamba and YOLOv8 networks, and features of the colonoscopy polyp images are extracted, and the long-range dependency capture capability and convolution local feature extraction capability of Mamba are utilized to enhance the coarse-grained and fine-grained information mining capability of the target polyps, and the Mamba attention mechanism is utilized to reduce the computational complexity of the YOLOv8 model;
[0010] In the decoding part of the Mamba and YOLOv8 neural network models, the YOLOv8 C2F block is improved and the Mamba attention mechanism is added to obtain layer-by-layer high-level semantic information; the bounding box and confidence of the polyp target in the colonoscopy image are calculated by the YOLOv8 detection head;
[0011] The encoding part of the Mamba and YOLOv8 neural networks is specifically:
[0012] The patch embedding layer takes as input the image Divided into non-overlapping patches of size 4×4, these patches are mapped to a new dimension C to generate an embedded image ;
[0013] Normalize the embedded image using layer normalization;
[0014] The patch merging layer is used to reduce the computational complexity by reducing the height and width of the input features, while increasing the number of channels to enhance the expressiveness of polyp features; the number of channels in each stage of the model of the present invention is [C, 2C, 4C, 8C], and C is set to 64;
[0015] S4, inputting the preprocessed test sample set obtained in step S2 into the neural network model trained in step S3 to obtain the colonoscopy polyp image detection result;
[0016] Preferably, step S1 specifically includes: converting the dataset annotation information into VOC format, and randomly dividing it into a training set, a validation set, and a test set in a ratio of 8:1.5:0.5.
[0017] Preferably, step S2 includes data enhancement of the polyp image, image rotation angle: -45°<α< +45°, hue: -40°< Hue <+40°, and Gaussian noise is added to the RGB three channels, and the size of all polyp sample images after enhancement is unified to 640×640.
[0018] Preferably, the improvement of the C2F block is specifically as follows:
[0019] The input feature map is segmented in the channel dimension, and the original feature map is cut into two parts. The feature maps after channel segmentation enter the MedVSSM module for further processing;
[0020] After the MedVSSM module processes the feature maps, these processed feature maps will be spliced and fused with the original feature maps that have not been processed by the MedVSSM module.
[0021] Preferably, the MedVSSM module part is specifically:
[0022] Using the formula The image input to the module is segmented according to the channel dimension. The segmented image Include and ;in, It is a feature map with half the number of channels, which is used for local feature extraction and enhancement in the subsequent convolution path; The feature map with the other half of the channels is used for subsequent VSSM path global feature extraction enhancement;
[0023] In the convolution branch, using the formula , using successive convolutional layers to Modeling of local features;
[0024] In the VSSM branch, we first use the formula Image Perform layer normalization. Among them, is layer normalization; after layer normalization, the feature map is divided into two branches; branch one uses the formula The feature map passes through the linear layer and activation function; branch two uses the formula Further extract the global features of the image and capture the long-distance dependencies; It is the two-dimensional selective scanning module (SS2D) of VMamba. is a depth-wise separable convolution, is a linear layer; finally, the formula is used Multiply the output feature map by the output of branch 1 element by element, that is, merge the feature maps of the two branches. It is an element-by-element multiplication operation; using the formula Perform feature mixing and add residual connections to obtain the output of the MedVSSM module;
[0025] After obtaining the output of the convolution branch and the VSSM branch, use the formula At the end of the network, the outputs of the two branches are merged along the channel dimension of the feature map, and the information interaction between channels is promoted; is a convolutional layer with a convolution kernel size of 1×1, It is the concat operation.
[0026] The beneficial effects of the present invention are as follows: the present invention is based on the attention mechanism and convolution local feature extraction characteristics of capturing long-distance dependencies of images based on the linear complexity of Mamba, and is based on the network framework of YOLOv8. In the Backbone part, by combining Mamba with convolution, local features and global features of the image are gradually extracted, and the local features extracted from the image are fused with the global features. In the Neck part, the fused features are upsampled, and feature maps of different sizes of Backbone are fused, so that the model learns deeper and more comprehensive features of the image. The model of the present invention can effectively capture the coarse-grained features and fine-grained features of polyps, and improve the accuracy of polyp detection. It is worth noting that due to the low computational complexity of Mamba, the present invention greatly reduces the number of parameters and the amount of calculation compared to the original YOLOv8 model, and the performance is still better than YOLOv8, which meets the needs of clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a flow chart of the present invention;
[0028] Figure 2 It is the overall framework diagram of the model of the present invention;
[0029] Figure 3It is a processing flow chart of the Backbone module in the Mamba and YOLO combination model of the present invention;
[0030] Figure 4 It is a schematic diagram of the structure of the MedVSSM module in the present invention;
[0031] Figure 5 Schematic diagram of the structure of the C2F-MedVSSM module in the present invention. DETAILED DESCRIPTION
[0032] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples.
[0033] See also Figure 1 , Figure 1 The flowchart of the present invention is as follows. A colonoscopy polyp image detection method based on Mamba and YOLOv8 in the present invention comprises the following steps:
[0034] S1, divide the acquired colonoscopy polyp image dataset into a training sample set, a validation sample set and a test sample set. Specifically:
[0035] S 11 ,This embodiment selects two public and challenging colonoscopy polyp image datasets (CVC-ClinicDB, Kvasir) as experimental datasets.
[0036] S 12 The two obtained colonoscopy polyp image data sets are randomly divided into training sample set, verification sample set, and test sample set in the ratio of 8.0:1.5:0.5, and then the divided training sample set, verification sample set, and test sample set are merged respectively. Finally, various forms of polyp image sample sets from different sampling environments are obtained. The sets are independent of each other and there are no identical pictures to prevent data contamination.
[0037] S 13 , transform the dataset annotation information into VOC format.
[0038] S2, perform data preprocessing operations on the sample set divided in step S1. Specifically:
[0039] S 21 ,The original resolution polyp images in the sample set divided in step S2 are uniformly adjusted to a size of 640×640 to facilitate training.
[0040] S 22 , for step S 21After resizing, each sample was augmented with data, including image rotation angle: -45°<α< +45°, hue: -40°< Hue <+40°, and Gaussian noise added to the RGB channels to simulate actual clinical colonoscopy acquisition conditions.
[0041] S3, using the pre-processed training sample set and verification sample set obtained in step S2 to train and verify the neural network model, and save the trained neural network model. Specifically:
[0042] S 31 The preprocessed training sample set obtained in step S2 is input into the Mamba and YOLOv8 neural network fusion model for training. The batch size is set to 16, the image input size is set to 640×640, the optimizer uses the SGD algorithm, the learning rate is set to 1e-2, the momentum is set to 0.937, and the weight decay exponent is set to 5e-4. The training process is stopped when the number of iterations reaches the set value (300) or the accuracy does not improve for 30 consecutive iterations. The parameter settings are the same as those of YOLOv8.
[0043] Further, see Figure 2 In the present invention, the Mamba and YOLOv8 neural network fusion model is an encoding-decoding-encoding architecture as a whole, specifically:
[0044] Backbone part: Patch embedding is performed on the image to divide the image into small blocks, and these small blocks are embedded in a high-dimensional space to convert them into the form of network input. The model then extracts multi-level features of the image through the MedVSSM module. After that, the features extracted by each MedVSSM module are fused through the patch fusion module to enhance the contextual information of the image and improve the expressiveness of the features. At the end of the Backbone, the feature map is subjected to spatial pyramid pooling to capture contextual information of different scales. By pooling feature maps at different scales, the model's ability to detect multi-scale targets is enhanced.
[0045] Neck part: The feature maps of different scales obtained from the Backbone part are upsampled and spliced by the C2F-MedVSSM module. The feature maps are upsampled to restore the low-resolution feature maps to images of the resolution size of the Backbone part, and the feature map information is integrated through splicing to learn the feature information of different levels of the image.
[0046] Head part: The feature map output by backbone and neck is sent to the detection head for object detection. The detection head will be responsible for generating the final detection box, category probability and other output information.
[0047] Further, see Figure 3 The Backbone calculation process in the present invention is as follows: in the Mamba and YOLOv8 neural network fusion model encoding part, the pre-processed training sample set obtained in step S2 is input into the model to capture the coarse and fine granularity information of the colonoscopic polyp image, which is used to model the long-range dependency relationship of the colonoscopic polyp image. Further, see Figure 2 , Figure 4 In the encoding part, the patch embedding layer first transforms the input image The image is divided into non-overlapping patches of size 4×4. These patches are then mapped to a new dimension C to generate an embedded image . In this way, the local features of the image can be expressed in a more compact way. Next, in order to further improve the stability and consistency of the polyp features, the embedded image is normalized using layer normalization. Subsequently, by stacking [2, 2, 6, 2] MedVSSM modules, the attention scanning mechanism of VSSM is used to capture the key information in the image, and the features are extracted and transformed through deep convolution and activation function operations. The patch merging layer is used to reduce the computational complexity by reducing the height and width of the input features, and the number of channels is increased to enhance the expression ability of the polyp features. The number of channels in each stage of the model of the present invention is [C, 2C, 4C, 8C], and C is set to 64.
[0048] Further, see Figure 4 The MedVSSM module in the present invention is specifically:
[0049] Using the formula Perform channel segmentation to segment the module input features along the channel dimension. Include and .in, It is a feature map with half the number of channels, which is used for local feature extraction and enhancement in the subsequent convolution path; The feature map with the other half of the channels is used for subsequent VSSM path global feature extraction enhancement;
[0050] The two sub-inputs are fed into the convolutional module and the SSM module branches respectively.
[0051] Using the formula ,Use continuous convolutional layers to model the local features of the input. Among them, It is implemented by three layers of convolution, and the sizes of the convolution kernels are 3×3, 3×3, and 1×1 respectively.
[0052] Using the formula Process the input of the SSM path, where is layer normalization. After layer normalization, the input is divided into two branches.
[0053] In the first branch of VSSM, using the formula Pass the input through a linear layer and an activation function.
[0054] In the second branch of the VSSM branch, using the formula Further feature extraction is performed to capture long-distance dependencies. It is the two-dimensional selective scanning module (SS2D) of VMamba. is a depth-wise separable convolution, is a linear layer.
[0055] Using the formula Multiply it element-wise with the output of the first branch to merge the two branches. It is an element-wise multiplication operation.
[0056] Using the formula Perform feature mixing and add residual connections to obtain the output of the MedVSSM module.
[0057] At the end of the network, using the formula The outputs of the two branches are merged along the channel dimension of the feature map, and the information interaction between channels is promoted. is a convolutional layer with a convolution kernel size of 1×1, It is the concat operation.
[0058] Further, see Figure 5 , the C2F-MedVSSM module in the present invention is specifically:
[0059] The core of the C2F-MedVSSM module is to enhance feature learning through channel segmentation and the MedVSSM module, while using residual connections to retain the original information. Specifically, the C2F-MedVSSM module first splits the input feature map in the channel dimension. This operation cuts the original feature map of size height×width×c_out into two parts, each with a channel number of 0.5×c_out, that is, two feature maps of size h×w×0.5c_out are obtained. The feature maps after channel segmentation enter the MedVSSM module for further processing. After the MedVSSM module processes the feature maps, these processed feature maps are spliced and fused with the original feature maps that have not been processed by the MedVSSM module. Residual connections help introduce new features while retaining the original feature information, thereby improving the network's expressiveness.
[0060] In addition, the C2F-MedVSSM module uses a serial connection method to connect multiple MedVSSM modules. These MedVSSM modules perform feature fusion through splicing operations, which is similar to integrating feature information at different levels. This serial connection and feature fusion method helps the network learn a more comprehensive and rich feature representation, allowing YOLOMamba to obtain richer gradient flow information while ensuring lightweight.
[0061] S 32 , input the preprocessed verification sample set obtained in step S2 into step S 31 In the trained Mamba and YOLOv8 neural network fusion model, the detection performance of the verification sample set is calculated using the mean average precision (mAP), a common indicator of target detection, and the relevant information of the Mamba and YOLOv8 neural network model parameters when the detection performance on the verification sample set is the best is saved.
[0062] S4, input the pre-processed test sample set obtained in step S2 into the neural network model trained in step S3 to obtain the colonoscopy polyp image detection result. Specifically:
[0063] S 41 The preprocessed test sample set obtained in step S2 is input into the trained neural network model in step S3 one by one to obtain the target border detection result of the colonoscopy polyp image and save it in JPG format.
Claims
1. A colonoscopy polyp image detection method based on Mamba and YOLOv8, characterized by: The following steps are involved: S1, obtain colonoscopic polyp image data and randomly divide it into training validation set and test set; S2, performs data preprocessing operations on the sample set divided by S1, including size standardization, rotation, hue change and noise addition operations; S3, using the pre-processed training sample set and verification sample set obtained in step S2 to train and verify the neural network model, and save the trained neural network model; specifically as follows: in the Mamba and YOLOv8 neural network model encoding part, the sample sets obtained in steps S1 and S2 are input into the Mamba and YOLOv8 networks, and features of the colonoscopy polyp images are extracted, and the long-range dependency capture capability and convolution local feature extraction capability of Mamba are utilized to enhance the coarse-grained and fine-grained information mining capability of the target polyps, and the Mamba attention mechanism is utilized to reduce the computational complexity of the YOLOv8 model; In the decoding part of the Mamba and YOLOv8 neural network models, the YOLOv8 C2F block is improved and the Mamba attention mechanism is added to obtain layer-by-layer high-level semantic information; the bounding box and confidence of the polyp target in the colonoscopy image are calculated by the YOLOv8 detection head; The encoding part of the Mamba and YOLOv8 neural networks is specifically: The patch embedding layer takes as input the image Divided into non-overlapping patches of size 4×4, these patches are mapped to a new dimension C to generate an embedded image ; Normalize the embedded image using layer normalization; Using the patch merging layer, the computational complexity is reduced by reducing the height and width of the input features, while increasing the number of channels to enhance the expressiveness of polyp features; the number of channels in each stage of the model is [C, 2C, 4C, 8C], and C is set to 64; S4, inputting the preprocessed test sample set obtained in step S2 into the neural network model trained in step S3 to obtain the colonoscopy polyp image detection result.
2. The colonoscopy polyp image detection method based on Mamba and YOLOv8 according to claim 1, characterized in that: Step S1 is specifically as follows: transform the dataset annotation information into VOC format, and randomly divide it into training set, validation set and test set in a ratio of 8:1.5:0.
5.
3. The colonoscopy polyp image detection method based on Mamba and YOLOv8 according to claim 1, characterized in that: Step S2 includes data enhancement of the polyp image, image rotation angle: -45°<α< +45°, hue: -40°< Hue<+40°, and Gaussian noise is added to the RGB three channels, and the size of all polyp sample images after enhancement is unified to 640×640.
4. The colonoscopy polyp image detection method based on Mamba and YOLOv8 according to claim 1, characterized in that: The improvements of the C2F block are specifically as follows: The input feature map is segmented in the channel dimension, and the original feature map is cut into two parts. The feature maps after channel segmentation enter the MedVSSM module for further processing; After the MedVSSM module processes the feature maps, these processed feature maps will be spliced and fused with the original feature maps that have not passed through the MedVSSM module.
5. The colonoscopy polyp image detection method based on Mamba and YOLOv8 according to claim 4, characterized in that: The MedVSSM module is specifically as follows: Using the formula Perform channel segmentation to segment the module input features along the channel dimension. Include and ;in, It is a feature map with half the number of channels, which is used for local feature extraction and enhancement in the subsequent convolution path; The feature map with the other half of the channels is used for subsequent VSSM path global feature extraction enhancement; In the convolution branch, using the formula , using successive convolutional layers to model local features of the input; In the VSSM branch, using the formula Process the input of the VSSM path, where is layer normalization; after layer normalization, the input is divided into two branches; using the formula Pass the input through the linear layer and activation function; using the formula Further feature extraction is performed to capture long-distance dependencies; among them, It is the two-dimensional selective scanning module (SS2D) of VMamba. is a depth-wise separable convolution, is a linear layer; using the formula Multiply it element-wise with the output of the first branch to merge the two branches; where It is an element-by-element multiplication operation; using the formula Perform feature mixing and add residual connections to obtain the output of the MedVSSM module; After getting the two branch outputs, use the formula At the end of the network, the outputs of the two branches are merged along the channel dimension of the feature map, and the information interaction between channels is promoted; is a convolutional layer with a convolution kernel size of 1×1, It is the concat operation.
