Intestinal polyp segmentation method and device, electronic equipment and storage medium
By combining pyramid pooled Transformer network, CNN network and wavelet-space higher-order feature aggregation network, multi-scale features are extracted and fused, and the boundary features of polyps are accurately learned, and the similarity of tissue characteristics, morphological diversity and intestinal polyps segmentation are solved. The challenge of boundary irregularity achieves more accurate and efficient intestinal polyps segmentation.
Patent Information
- Application Number
- CN202411277431.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2025-02-07
AI Technical Summary
In the diagnosis of colorectal cancer, the segmentation task of intestinal polyps faces challenges such as tissue characteristics similarity, morphological diversity and boundary irregularity, which leads to difficulties in accurate identification and segmentation.
Using pyramid pooled Transformer network, CNN network and wavelet-space higher-order feature aggregation network, multi-scale feature learning module and multi-scale feature aggregation module are used to extract and fuse multi-scale features to accurately learn the boundary features of polyps. In this way, more accurate intestinal polyps segmentation is achieved.
It improves the accuracy and efficiency of intestinal polyps segmentation, reduces the rate of missed diagnosis, enhances the ability to identify irregular polyps, and solves the problem of boundary uncertainty.
Smart Images
Figure CN119399212A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the technical field of medical image processing, and in particular to a method and device for segmenting intestinal polyps, an electronic device, and a storage medium. Background Art
[0002] Colorectal cancer (CRC) is one of the most common cancers that endangers human health. According to relevant data, colorectal cancer has become the world's third highest incidence (9.6%) and second highest mortality (9.3%) cancer. The pathogenesis of colorectal cancer is mainly caused by an unhealthy lifestyle that leads to further deterioration of polyps and finally develops into colorectal cancer. Therefore, early polyp treatment plays an important role in preventing the onset of colorectal cancer. As the gold standard for diagnosing colorectal diseases, colonoscopy has an irreplaceable position in clinical practice, and the problem of missed diagnosis in its diagnostic process cannot be ignored. In clinical practice, due to the differences in professional knowledge and experience of different experts in different polyp lesions, colonoscopy not only wastes time and manpower, but also may lead to a missed diagnosis rate of 15%-24%.
[0003] Polyp segmentation technology, as a branch in the field of medical image processing, distinguishes polyp areas from normal tissues through computer vision algorithms, which is of great significance for improving the diagnostic quality of colonoscopy. However, the polyp segmentation task faces a series of challenges in practical applications, such as (1) Similarity of tissue features: The difference in color and texture between the polyp area and the surrounding normal tissue is not obvious, resulting in low contrast and increasing the difficulty of identification. (2) Diversity of morphology: The diversity of size, color and morphology of polyps at different stages brings complexity to accurate identification. (3) Irregularity of boundaries: The irregular shape and fuzzy boundaries of polyps, especially the similarity with the folds of the intestinal wall, make accurate segmentation particularly difficult. Therefore, how to achieve more accurate and efficient intestinal polyp segmentation has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The embodiments of the present invention provide a method and device for segmenting intestinal polyps, an electronic device and a storage medium, which can achieve more accurate and efficient intestinal polyp segmentation.
[0005] In a first aspect, an embodiment of the present invention provides a method for segmenting intestinal polyps, comprising:
[0006] Obtain images of intestinal polyps;
[0007] Preprocessing the intestinal polyp image to obtain a standardized intestinal polyp image;
[0008] Extracting features from the standardized intestinal polyp image to obtain multiple image features at different scales;
[0009] The image features of multiple different scales are input into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn the image features of multiple different scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on the image features of multiple different scales, the boundary features and the supplementary features to output a segmentation map.
[0010] In some embodiments, the training method of the intestinal polyp segmentation model includes:
[0011] Acquire an intestinal polyp image dataset, wherein the intestinal polyp image dataset includes labeled intestinal polyp segmentation data;
[0012] The intestinal polyp segmentation model is trained using the labeled intestinal polyp segmentation data, and during the training process, the model parameters are optimized by a back propagation algorithm to obtain target model parameters;
[0013] The trained intestinal polyp segmentation model is obtained based on the target model parameters.
[0014] In some embodiments, the preprocessing of the intestinal polyp image to obtain a standardized intestinal polyp image includes:
[0015] Performing a segmentation operation on the intestinal polyp image to obtain segmented image slices;
[0016] The segmented image slices are labeled to obtain the standardized intestinal polyp image.
[0017] In some embodiments, the boundary learning module includes discrete wavelet transform, deformable convolution, spatial attention layer, global maximum pooling layer, global average pooling layer, element addition, gated fusion, average pooling and maximum pooling. The boundary learning module performs average pooling and maximum pooling on the high-level features in the image features, respectively, performs gated fusion operations on the high-level features after average pooling and maximum pooling, uses deformable convolution to convolve the high-level features after gated fusion and then inputs them into the spatial attention layer, and then performs deformable convolution again. Each convolution operation adopts a residual connection method, and the output boundary features and the low-level features in the image features are subjected to the above operation again.
[0018] In some embodiments, the global maximum pooling layer and the global average pooling layer are used to extract the maximum value and the average value, respectively, within the global range of the image features.
[0019] In some embodiments, the multi-scale feature learning module includes deformable convolution, wavelet transform and depth-separable convolution. The multi-scale feature learning module divides the image features of multiple scales into four parallel branches for convolution operations. Each branch is combined using convolution layers of different types and sizes. After the convolution operation of the four parallel branches, a splicing operation is used to splice the features of the four branches according to the channel dimension, and finally a wavelet transform is performed to output supplementary features of different frequencies.
[0020] In some embodiments, the multi-scale feature aggregation module includes deformable convolution, wavelet transform, and inverse wavelet transform. The multi-scale feature aggregation module first performs three wavelet transforms on the low-level features in the image features, and each time the size of the wavelet transform feature map is halved and the channel is doubled. Then, each time the wavelet transform feature accepts and fuses the boundary features output by the boundary learning module and the supplementary features output by the multi-scale feature learning module, and deformable convolution and inverse wavelet transform operations are performed on the boundary features and the supplementary features in turn, wherein the deformable convolution halves the number of channels of the feature map, and the inverse wavelet transform doubles the size of the feature map. The above operations are performed step by step and finally a segmentation map is output.
[0021] In a second aspect, an embodiment of the present invention further provides an intestinal polyp segmentation device, the device comprising:
[0022] An acquisition module, used for acquiring intestinal polyp images;
[0023] A processing module, used for preprocessing the intestinal polyp image to obtain a standardized intestinal polyp image;
[0024] An extraction module, used for extracting features from the standardized intestinal polyp image to obtain image features of multiple different scales;
[0025] A segmentation module is used to input the image features of multiple different scales into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn the image features of multiple different scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on the image features of multiple different scales, the boundary features and the supplementary features to output a segmentation map.
[0026] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the intestinal polyp segmentation method as described in the first aspect when executing the computer program.
[0027] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the intestinal polyp segmentation method as described in the first aspect.
[0028] According to the intestinal polyp segmentation method and device, electronic device and storage medium provided by the embodiments of the present invention, the intestinal polyp segmentation method includes: obtaining an intestinal polyp image; preprocessing the intestinal polyp image to obtain a standardized intestinal polyp image; extracting features of the standardized intestinal polyp image to obtain image features of multiple scales; inputting the image features of multiple scales into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn image features of multiple scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on image features of multiple scales, boundary features and supplementary features to output a segmentation map. The intestinal polyp segmentation model of the present invention introduces a combination of pyramid pooling Transformer (Pyramid Pooling Transformer, P2T) and CNN, and adds a wavelet-space high-order feature aggregation network. In the first stage, a multi-scale feature learning module is used to learn multiple features of different scales output by P2T so that the network can better learn local supplementary features, which are used to supplement features when model features are fused. In the second stage, a boundary learning module is used to better learn the boundary features of polyps to avoid image noise interference. The learned boundary features are used to perform boundary constraints on the output segmentation map at the multi-scale feature aggregation module, thereby solving the problem of boundary uncertainty in the field of polyp segmentation. In the third stage, a multi-scale feature aggregation module is used to fuse features of different scales to better output the segmentation map. Based on this, the embodiment of the present invention can achieve more accurate and efficient intestinal polyp segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 is a flow chart of an intestinal polyp segmentation method provided by one embodiment of the present invention;
[0030] Figure 2 This is an overall framework diagram of an intestinal polyp segmentation model provided by an embodiment of the present invention;
[0031] Figure 3 is a framework diagram of a boundary learning module provided by an embodiment of the present invention;
[0032] Figure 4 is a framework diagram of a multi-scale feature learning module provided by an embodiment of the present invention;
[0033] Figure 5 is a framework diagram of a multi-scale feature aggregation module provided by an embodiment of the present invention;
[0034] Figure 6 is a schematic structural diagram of an intestinal polyp segmentation device provided by an embodiment of the present invention;
[0035] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0037] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the following drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0038] In the embodiments of the present invention, the words "further", "exemplarily" or "optionally" are used to indicate examples, illustrations or descriptions, and should not be interpreted as being more preferred or more advantageous than other embodiments or designs. The use of the words "further", "exemplarily" or "optionally" is intended to present related concepts in a specific way.
[0039] First, several terms involved in the present invention are analyzed:
[0040] Intestinal polyps: These are small fleshy growths that grow on the lining of the intestine, usually round or oval in shape. Polyps can be benign, but can sometimes develop into colorectal cancer.
[0041] Intestinal polyp segmentation: Intestinal polyp segmentation is a method in the field of medical image processing, which refers to the automatic identification and segmentation of intestinal polyp areas in images obtained by colonoscopy or other medical imaging technologies.
[0042] Colonoscope: A medical device used to examine the inside of the colon and rectum. It helps doctors see inside the intestines and look for polyps, ulcers, inflammation or other abnormalities, and is an important tool for diagnosing colon and rectal diseases.
[0043] Colonoscopy: A colonoscopy is a type of endoscopy in which a soft, flexible tube is inserted through the anus, allowing the doctor to view the entire colon and rectum. This test is very effective in detecting polyps, cancer, or other bowel diseases. Before the test, patients are usually required to do a bowel preparation, which includes taking a laxative to empty the intestines, and possibly fasting.
[0044] CNN: Convolutional Neural Network.
[0045] Transformer: Similar to convolutional neural networks, it is a deep learning model based on the self-attention mechanism (Se l f-Attention Mechan is m), which has achieved great success in the field of natural language processing (NLP), especially in tasks such as language translation, text summarization, and question-answering systems.
[0046] In order to more conveniently describe the working principle of the embodiment of the present invention later, an introduction to relevant technical scenarios is first given below.
[0047] Colorectal cancer (CRC) is one of the most common cancers that endangers human health. According to relevant data, colorectal cancer has become the world's third highest incidence (9.6%) and second highest mortality (9.3%) cancer. The pathogenesis of colorectal cancer is mainly caused by an unhealthy lifestyle that leads to further deterioration of polyps and finally develops into colorectal cancer. Therefore, early polyp treatment plays an important role in preventing the onset of colorectal cancer. As the gold standard for diagnosing colorectal diseases, colonoscopy has an irreplaceable position in clinical practice, and the problem of missed diagnosis in its diagnostic process cannot be ignored. In clinical practice, due to the differences in professional knowledge and experience of different experts in different polyp lesions, colonoscopy not only wastes time and manpower, but also may lead to a missed diagnosis rate of 15%-24%.
[0048] Polyp segmentation technology, as a branch in the field of medical image processing, distinguishes polyp areas from normal tissues through computer vision algorithms, which is of great significance for improving the diagnostic quality of colonoscopy. However, the polyp segmentation task faces a series of challenges in practical applications, such as (1) Similarity of tissue features: The difference in color and texture between the polyp area and the surrounding normal tissue is not obvious, resulting in low contrast and increasing the difficulty of identification. (2) Diversity of morphology: The diversity of size, color and morphology of polyps at different stages brings complexity to accurate identification. (3) Irregularity of boundaries: The irregular shape and fuzzy boundaries of polyps, especially the similarity with the folds of the intestinal wall, make accurate segmentation particularly difficult. Therefore, how to achieve more accurate and efficient intestinal polyp segmentation has become a technical problem that needs to be solved urgently.
[0049] Based on this, the present invention provides a method and device for intestinal polyp segmentation, an electronic device and a storage medium. The intestinal polyp segmentation method includes: obtaining an intestinal polyp image; preprocessing the intestinal polyp image to obtain a standardized intestinal polyp image; extracting features from the standardized intestinal polyp image to obtain image features of multiple scales; inputting the image features of multiple scales into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn image features of multiple scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on image features, boundary features and supplementary features of multiple scales to output a segmentation map. The intestinal polyp segmentation model of the present invention introduces a combination of pyramid pooling Transformer (Pyramid Pooling Transformer, P2T) and CNN, and adds a wavelet-space high-order feature aggregation network. In the first stage, a multi-scale feature learning module is used to learn multiple features of different scales output by P2T so that the network can better learn local supplementary features, which are used to supplement features when model features are fused. In the second stage, a boundary learning module is used to better learn the boundary features of polyps to avoid image noise interference. The learned boundary features are used to perform boundary constraints on the output segmentation map at the multi-scale feature aggregation module, thereby solving the problem of boundary uncertainty in the field of polyp segmentation. In the third stage, a multi-scale feature aggregation module is used to fuse features of different scales to better output the segmentation map. Based on this, the embodiment of the present invention can achieve more accurate and efficient intestinal polyp segmentation.
[0050] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.
[0051] like Figure 1 As shown, Figure 1 It is a flow chart of an intestinal polyp segmentation method provided by an embodiment of the present invention. The intestinal polyp segmentation method may include but is not limited to steps S101 to S104.
[0052] Step S101, obtaining an intestinal polyp image;
[0053] Step S102, preprocessing the intestinal polyp image to obtain a standardized intestinal polyp image;
[0054] Step S103, extracting features from the standardized intestinal polyp image to obtain image features of multiple different scales;
[0055] Step S104, inputting multiple image features of different scales into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn image features of multiple different scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on image features, boundary features and supplementary features of multiple different scales to output a segmentation map.
[0056] It can be understood that the intestinal polyp segmentation model of the present invention, on the premise of introducing the combination of pyramid pooling Transformer (Pyramid Pooling Transformer, P2T) and CNN, adds a wavelet-space high-order feature aggregation network (WHFA-Net). In the first stage, a multi-scale feature learning module is used to learn the features of multiple different scales output by P2T so that the network can better learn local supplementary features, which are used to supplement features when the model features are fused. In the second stage, a boundary learning module is used to better learn the boundary features of polyps to avoid image noise interference. The learned boundary features are used to output the boundary constraints of the segmentation map at the multi-scale feature aggregation module, thereby solving the problem of boundary uncertainty in the field of polyp segmentation. In the third stage, a multi-scale feature aggregation module is used to fuse features of different scales to better output the segmentation map. Based on this, the embodiment of the present invention can achieve more accurate and efficient intestinal polyp segmentation.
[0057] It should be noted that the intestinal polyp segmentation method of the embodiment of the present invention can be used to segment and extract the pathological parts of intestinal polyps captured by colonoscopy, helping doctors improve diagnostic efficiency and accuracy in clinical practice. The intestinal images captured by the colonoscope are subjected to feature extraction and learning based on the model combined with CNN and Transformer to segment intestinal polyps, helping doctors to more accurately and efficiently screen out the polyp parts that are not easy to distinguish during colonoscopy. For example, small polyps, irregular polyps, and polyps that are not easy to distinguish. This method can be applied to the colonoscopy auxiliary system to provide diagnostic suggestions for doctors in clinical practice.
[0058] It is understandable that if Figure 2 As shown, Figure 2The figure is the overall framework diagram of the intestinal polyp segmentation model. The intestinal polyp segmentation model includes a boundary learning module A, a multi-scale feature learning module B and a multi-scale feature aggregation module C. The boundary learning module A is used to learn the boundary and is supervised by the external loss. The multi-scale feature learning module B is used to better learn features of different scales to supplement the features when the model features are fused. The multi-scale feature aggregation module C is used to fuse the learned features of different scales and finally output the segmentation map.
[0059] It is understandable that if Figure 3 As shown, Figure 3 The figure is the framework diagram of the boundary learning module A. The boundary learning module A contains a network of discrete wavelet transform (DWT), deformable convolution (Deconv), spatial attention (SA), global maximum pooling (GMP), global average pooling (GAP), element-wise addition, gated fusion, average pooling and max pooling. The following is a step-by-step description of the content of the boundary learning module A: The boundary learning module A receives two high-level features for maximum pooling and average pooling respectively. The global maximum pooling (GMP) and global average pooling (GAP) layers are used to extract the maximum value and average value in the global range of the feature map, which helps the model understand the global context information. Then the features after average pooling and maximum pooling are gated for fusion. The gated fusion features are convolved with deformable convolution and then input into the spatial attention module. The spatial attention (SA) layer can be used to enhance the network's attention to important areas, which helps the model better focus on the key parts of the image. Then another deformable convolution is performed. Each convolution operation uses a residual connection method. After that, the output boundary features and the lowest level features are subjected to the above operation again.
[0060] It is understandable that if Figure 4 As shown, Figure 4The framework diagram of the multi-scale feature learning module B. The multi-scale feature learning module B contains a network of components such as deformable convolution (DeformConv), wavelet transform (WT), and depthwise separable convolution (Depthwise Separable Convolution). The following is a step-by-step description of the content of the multi-scale feature learning module B: First, the input features are divided into four parallel branches for convolution operations. Each branch is combined with convolution layers of different types and sizes to improve the ability to capture local features of the image. After the convolution operation of the four branches, the features of the four branches are spliced according to the channel dimension using a splicing operation. Finally, a wavelet transform is performed to output features of different frequencies.
[0061] It should be noted that for the multi-scale feature learning module B, external branches can be added on the basis of the above. The multi-branch structure can learn different features and representations through each branch, thereby improving the learning ability of the system.
[0062] It is understandable that if Figure 5 As shown, Figure 5 The framework diagram of the multi-scale feature aggregation module C. The multi-scale feature aggregation module C is a network containing deformable convolution (DeformConv), wavelet transform (WT), and inverse wavelet transform (Inverse Wavelet Transform) components. The following is a step-by-step description of the content of the multi-scale feature aggregation module C: The multi-scale feature aggregation module C first performs three wavelet transforms on the lowest-level features, and each time the size of the wavelet transform feature map is halved and the number of channels is doubled. Then, each wavelet transform feature accepts and fuses the features output by the boundary learning module A and the multi-scale feature learning module B. After receiving the features output by the boundary learning module A and the multi-scale feature learning module B, deformable convolution and inverse wavelet transform operations are performed in sequence. Among them, the deformable convolution halves the number of channels in the feature map, and the inverse wavelet transform doubles the size of the feature map. The above operations are performed step by step and finally the segmentation map is output.
[0063] It should be noted that for the multi-scale feature aggregation module C, the multi-scale feature aggregation module C can also use a graph neural network (GNN) to explore the correlation between pixels. In a graph neural network, a graph is represented as a collection of nodes and edges, where nodes represent entities and edges represent the relationship between entities. The relationship between pixels can be regarded as the attributes associated with the nodes. However, in some cases, the graph data may be very sparse, that is, there are few connections between nodes. This may make it difficult for the graph neural network to capture the relationship information between nodes.
[0064] In summary, the wavelet-spatial high-order feature aggregation network (WHFA-Net) consists of three main stages. The first stage is to use the multi-scale feature learning module B to learn the features of four different scales output by P2T so that the network can better learn local features. The second stage is to use the boundary learning module A to learn the boundary part of the polyp, which is used to output the boundary constraints of the segmentation map at the multi-scale feature aggregation module C. The third stage is to use the multi-scale feature aggregation module C to fuse the features of different scales to better output the segmentation map.
[0065] Based on this, the present invention introduces a pyramid pooling Transformer and combines it with CNN, and adds wavelet transform technology to construct a new segmentation method (WHFA-Net (wavelet-space high-order feature aggregation network)). Wavelet transform can make full use of features and avoid information loss and information conflict in the process of feature fusion at different levels. Specifically, boundary learning is performed by combining wavelet transform with CNN to perform boundary constraints during feature fusion to solve the problem of unclear boundaries between lesion areas and non-lesion areas. Wavelet transform is used for feature fusion in the decoding stage to solve the problems of semantic conflicts and information redundancy in the decoding information loss and feature fusion process.
[0066] In one embodiment, the intestinal polyp segmentation method based on CNN and Transformer of the present invention can be used to assist doctors in identifying and extracting polyp lesion areas. The method combines CNN and Transformer models to achieve more accurate and efficient intestinal polyp segmentation. The main steps of the method are as follows:
[0067] 1) Data collection: Collect and organize a large number of intestinal polyp image datasets.
[0068] 2) Preprocessing: The collected image data is preprocessed. Preprocessing includes intestinal polyp segmentation and annotation of the image to prepare standardized image data for input into the model.
[0069] 3) Feature extraction: The preprocessed feature data is input into the basic model for simple feature extraction. Then, one branch performs boundary learning, and the other branch performs feature fusion of four scales. In the process of feature fusion, the features learned from the boundary are also integrated.
[0070] 4) Intestinal polyp segmentation: After the feature fusion is completed, the segmentation map predicted by the model is output.
[0071] 5) Model training and optimization: The model is trained using labeled intestinal polyp segmentation data, and the model parameters are optimized through the back propagation algorithm to improve segmentation accuracy and generalization ability.
[0072] It should be noted that the model combining Transformer and CNN is used for the intestinal polyp segmentation model, so that the intestinal polyp segmentation model captures both local features and global features, and outputs a better segmentation map in the feature fusion stage.
[0073] It should be noted that the intestinal polyp segmentation model also uses a wavelet-space high-order feature aggregation network, which includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module. The boundary learning module is used to learn the boundary features of intestinal polyps, and the multi-scale feature learning module is used to learn multiple image features of different scales to obtain supplementary features. The multi-scale feature aggregation module is used to fuse the image features, boundary features and supplementary features of multiple different scales to output a segmentation map. The intestinal polyp segmentation model of the present invention is based on the introduction of the pyramid pooling Transformer and CNN, and adds a wavelet-space high-order feature aggregation network. In the first stage, the multi-scale feature learning module is used to learn the multiple different scales of features output by P2T so that the network can better learn the local supplementary features, which are used to supplement the features when the model features are fused. In the second stage, the boundary learning module is used to better learn the boundary features of the polyps to avoid image noise interference. The learning boundary features are used to output the boundary constraints of the segmentation map at the multi-scale feature aggregation module, thereby solving the problem of boundary uncertainty in the field of polyp segmentation. In the third stage, the multi-scale feature aggregation module is used to fuse the features of different scales to better output the segmentation map.
[0074] Based on this, compared with the prior art, the intestinal polyp segmentation method of the present invention has at least the following beneficial effects:
[0075] (1) Comprehensive feature learning: The combination of CNN and Transformer can make full use of their respective characteristics. CNN can extract local features in the convolution layer, while Transformer can capture global dependencies in the self-attention mechanism, thereby better learning the long-range dependencies between features. By combining CNN and Transformer, a more comprehensive and accurate feature representation can be obtained, improving the performance of intestinal polyp segmentation.
[0076] (2) Boundary constraints: A separate boundary learning module can better learn boundary features and avoid image noise interference.
[0077] (3) Better multi-scale feature fusion: Wavelet transform and inverse wavelet transform operations can better preserve the original image information during feature downsampling and upsampling, thereby avoiding information loss during feature fusion.
[0078] In addition, if Figure 6As shown, one embodiment of the present invention further discloses an intestinal polyp segmentation device, the device comprising:
[0079] An acquisition module 110 is used to acquire an image of intestinal polyps;
[0080] The processing module 120 is used to pre-process the intestinal polyp image to obtain a standardized intestinal polyp image;
[0081] An extraction module 130 is used to extract features from the standardized intestinal polyp image to obtain image features of multiple different scales;
[0082] The segmentation module 140 is used to input image features of multiple different scales into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn image features of multiple different scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on image features, boundary features and supplementary features of multiple different scales to output a segmentation map.
[0083] The intestinal polyp segmentation device of the embodiment of the present invention is used to execute the intestinal polyp segmentation method in the above embodiment. Its specific processing process is the same as the intestinal polyp segmentation method in the above embodiment, and will not be repeated here.
[0084] In addition, if Figure 7 As shown, an embodiment of the present invention further discloses an electronic device, comprising: at least one processor 210; at least one memory 220, for storing at least one program; when the at least one program is executed by the at least one processor 210, the intestinal polyp segmentation method as in any of the previous embodiments is implemented.
[0085] In addition, an embodiment of the present invention further discloses a computer-readable storage medium, in which computer-executable instructions are stored. The computer-executable instructions are used to execute the intestinal polyp segmentation method in any of the previous embodiments.
[0086] The system architecture and application scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of the system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.
[0087] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0088] In hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those of ordinary skill in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0089] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process or an execution thread, and a component may be located on a computer or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may communicate, for example, through local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems through signals).
Claims
1. A method for segmenting intestinal polyps, comprising: Obtain images of intestinal polyps; Preprocessing the intestinal polyp image to obtain a standardized intestinal polyp image; Extracting features from the standardized intestinal polyp image to obtain multiple image features at different scales; The image features of multiple different scales are input into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn the image features of multiple different scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on the image features of multiple different scales, the boundary features and the supplementary features to output a segmentation map.
2. The method according to claim 1, characterized in that The training method of the intestinal polyp segmentation model includes: Acquire an intestinal polyp image dataset, wherein the intestinal polyp image dataset includes labeled intestinal polyp segmentation data; The intestinal polyp segmentation model is trained using the labeled intestinal polyp segmentation data, and during the training process, the model parameters are optimized by a back propagation algorithm to obtain target model parameters; The trained intestinal polyp segmentation model is obtained based on the target model parameters.
3. The method according to claim 1, characterized in that The preprocessing of the intestinal polyp image to obtain a standardized intestinal polyp image includes: Performing a segmentation operation on the intestinal polyp image to obtain segmented image slices; The segmented image slices are labeled to obtain the standardized intestinal polyp image.
4. The method according to claim 1, characterized in that: The boundary learning module includes discrete wavelet transform, deformable convolution, spatial attention layer, global maximum pooling layer, global average pooling layer, element addition, gated fusion, average pooling and maximum pooling. The boundary learning module performs average pooling and maximum pooling on the high-level features in the image features respectively, performs gated fusion operation on the high-level features after average pooling and maximum pooling, uses deformable convolution to convolve the high-level features after gated fusion and then inputs them into the spatial attention layer, and then performs deformable convolution again. Each convolution operation adopts a residual connection method, and the output boundary features and the low-level features in the image features are subjected to the above operation again.
5. The method according to claim 4, characterized in that The global maximum pooling layer and the global average pooling layer are used to extract the maximum value and the average value respectively in the global range of the image features.
6. The method according to claim 1, characterized in that The multi-scale feature learning module includes deformable convolution, wavelet transform and depth-separable convolution. The multi-scale feature learning module divides the image features of multiple scales into four parallel branches for convolution operations. Each branch is combined using convolution layers of different types and sizes. After the convolution operation of the four parallel branches, the features of the four branches are spliced according to the channel dimension using a splicing operation, and finally a wavelet transform is performed to output supplementary features of different frequencies.
7. The method according to claim 1, characterized in that The multi-scale feature aggregation module includes deformable convolution, wavelet transform, and inverse wavelet transform. The multi-scale feature aggregation module first performs three wavelet transforms on the low-level features in the image features. Each time the wavelet transform feature map is halved in size and the channels are doubled. Then, each wavelet transform feature accepts and fuses the boundary features output by the boundary learning module and the supplementary features output by the multi-scale feature learning module. The boundary features and the supplementary features are sequentially subjected to deformable convolution and inverse wavelet transform operations, wherein the deformable convolution halves the number of channels in the feature map, and the inverse wavelet transform doubles the size of the feature map. The above operations are performed step by step and finally a segmentation map is output.
8. An intestinal polyp segmentation device, characterized in that: The device comprises: An acquisition module, used for acquiring intestinal polyp images; A processing module, used for preprocessing the intestinal polyp image to obtain a standardized intestinal polyp image; An extraction module, used for extracting features from the standardized intestinal polyp image to obtain image features of multiple different scales; A segmentation module is used to input the image features of multiple different scales into a pre-trained intestinal polyp segmentation model to obtain an intestinal polyp segmentation result, wherein the intestinal polyp segmentation model includes a pyramid pooling Transformer network, a CNN network and a wavelet-space high-order feature aggregation network, the wavelet-space high-order feature aggregation network includes a boundary learning module, a multi-scale feature learning module and a multi-scale feature aggregation module, the boundary learning module is used to learn the boundary features of intestinal polyps, the multi-scale feature learning module is used to learn the image features of multiple different scales to obtain supplementary features, and the multi-scale feature aggregation module is used to perform feature fusion on the image features of multiple different scales, the boundary features and the supplementary features to output a segmentation map.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the intestinal polyp segmentation method as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the intestinal polyp segmentation method according to any one of claims 1 to 7.
Citation Information
Cited By
Endoscopic image polyp segmentation method and device, computer equipment and storage medium
CN120823382A