An intestinal polyp image segmentation method simulating polyp growth

Abstract

The present application belongs to the technical field of deep learning in computer vision, and provides an intestinal polyp image segmentation method for simulating polyp growth. The present application decouples the complete ground truth map into a Gaussian ground truth map and a main body ground truth map to pay more attention to position information and main body information respectively; three feature extraction branches are constructed to extract features, and are supervised by the Gaussian ground truth map, the main body ground truth map and the complete ground truth map respectively; and a dynamic attention guidance module is designed to more efficiently aggregate features from different branches. The present application realizes a more accurate polyp segmentation model, which can accurately segment polyps of various complex shapes and backgrounds.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and involves deep learning and computer vision, particularly a method for segmenting intestinal polyp images to simulate polyp growth. Background Technology

[0002] Colorectal cancer poses a serious threat to human health. It typically develops gradually from small polyps in the intestines; therefore, the best way to prevent colorectal cancer is to accurately identify these polyps early on. Currently, colonoscopy is the most common method, but this process requires doctors to carefully search for polyps on a screen, which is not only time-consuming and laborious but also has a high rate of missed diagnoses. Therefore, it is crucial to introduce computer vision technology into the field of intestinal polyp segmentation and design an accurate and efficient automated polyp diagnostic system to assist doctors in diagnosis.

[0003] Research methods for intestinal polyp segmentation mainly fall into two categories: traditional methods and deep learning-based methods. Traditional methods primarily utilize heuristic priors, such as boundary point priors, or human intuition, such as background comparison or color comparison. Traditional methods involve manual feature extraction, which is time-consuming, labor-intensive, and generally yields poor detection results.

[0004] In recent years, the rapid development of deep learning has greatly promoted the field of polyp segmentation, leading to the proposal of many new deep learning-based methods. Unlike traditional methods, deep learning-based polyp segmentation methods can automatically learn the features of the target without manual extraction. More importantly, the performance of deep learning-based intestinal polyp segmentation methods has been significantly improved, with lighter models, faster processing speeds, and substantial increases in accuracy. Although the field of intestinal polyp segmentation has made great strides, there is still considerable room for improvement. For example, detection performance degrades significantly when facing complex backgrounds or multi-scale problems, and real-time performance still has significant room for improvement. Summary of the Invention

[0005] The technical problem to be solved by this invention is to make up for the shortcomings of current polyp image segmentation tasks and propose an intestinal polyp image segmentation method that simulates polyp growth to achieve high-precision and high-efficiency segmentation.

[0006] The technical solution of the present invention:

[0007] A method for segmenting intestinal polyp images by simulating polyp growth employs a polyp growth simulation network. This network utilizes a deep learning model and primarily consists of a basic feature extraction network, a feature extraction and fusion module, and a dynamic attention guidance module. The basic feature extraction network extracts multi-level features from the intestinal polyp image through an encoder. The feature extraction and fusion module further integrates and enhances the original features extracted by the encoder to output a predicted image. This module includes three branches: a Gaussian branch, a main body branch, and a complete polyp prediction branch. The dynamic attention guidance module more efficiently integrates information from these three branches. The specific steps are as follows:

[0008] Step (1) Constructing the basic feature extraction network

[0009] The feature extraction network uses an encoder to extract multi-level feature information from the input image. The feature extraction network uses a PVT network pre-trained on ImageNet as the encoder. Its input is a 3-channel RGB image, and its output is multi-level features f1, f2, f3, and f4, where f1 is a low-level feature, f2 is a mid-level feature, and f3 and f4 are high-level features.

[0010] (2) Constructing a feature extraction and fusion module

[0011] The feature extraction and fusion module has three branches: the Gaussian branch, the main body branch, and the complete polyp prediction branch. These branches fuse features at different levels and are supervised by the Gaussian ground truth map, the main body ground truth map, and the complete polyp prediction map, respectively, as detailed below:

[0012] Gaussian branch: Using only high-level features f4 and f3, the output feature f of the Gaussian branch is finally obtained. G The fusion process is shown in the following formula:

[0013] f G =F(Cat(F(up(f4))·f3,F(up(f4))))

[0014] Where · represents element-wise multiplication, up(·) represents upsampling, Cat(·) represents concatenation, and F(·) represents a combination of convolution, batch normalization, and ReLU activation functions.

[0015] Main branch: Using high-level features f4, f3 and intermediate feature f2; first, aggregate the three features to form a richer combined feature f′:

[0016] f′=F(F(f4)·f3)·f2

[0017] Then, a Dynamic Attention Guidance Module (DAG) is used to further achieve feature fusion, obtaining the output feature f of the main branch. B :

[0018] f B =DAG(f G ,F(Cat(F(up(f4)),F(up(f3),f′)))

[0019] Full polyp prediction branch: using all features f4, f3, f2, and f1; first, the enhancement feature f″ is obtained:

[0020] f″=F(F(F(f4)·f3)·f2)·f1

[0021] Then, a Dynamic Attention Guidance Module (DAG) is used to further perform feature fusion, resulting in the final predicted features f. P :

[0022] f P =DAG(f B ,F(Cat(F(f4),F(f3),F(f2),f”)))

[0023] Finally, the results of the three branches are upsampled to the same size as the input image and supervised by three ground truth graphs respectively.

[0024] The dataset contains images of intestinal polyps and their corresponding complete ground truth maps. The Gaussian ground truth map and the main ground truth map need to be generated decoupled from the complete ground truth map, according to the following rules:

[0025] Generation of the Gaussian truth map: The complete truth map (denoted as I) is generated from the background pixels (I0). bg ) and foreground pixels (I fg It consists of two parts; first, the center point of the foreground pattern is located based on the complete truth map. Specifically, a rectangle is used to completely surround the foreground pattern, and the center point of the rectangle is taken as the center point p of the foreground pattern. c Its coordinates are represented as (x c y c The length of the shorter side of the rectangle is denoted as R. Then, the value of each coordinate point p in the diagram is calculated using the formula and denoted as O′(p), where the coordinates of p are represented as (x, y).

[0026]

[0027] Finally, the Gaussian ground truth map O is obtained according to the following formula. Gaussian :

[0028]

[0029] Where p∈I represents all coordinate points in the complete truth graph.

[0030] Generation of the main truth map: First, define f(p, q) as the Euclidean distance between two pixels p and q, I fg I represents the foreground pixel. bg This represents the background pixels. Then, the value O″(p) of each pixel is calculated using the formula:

[0031]

[0032] Normalize the obtained O″(p) to obtain the main truth map O. body .

[0033] Step (3) Construct the Dynamic Attention Guidance Module (DAG)

[0034] Fuse guided features f using a dynamic attention guidance module. a and features f with rich semantic information b Specifically, in the network structure, modules are placed in different locations, and their f a and f b Different specific features can be incorporated. In the main branch, f a f G f b For f4, f3, and f2; in the complete polyp prediction branch, f a f B f b Let the features be f4, f3, f2, and f1. First, the two features f a and f b f is generated by implementing linear mapping through 1*1 convolution. a1 f b1 and f b2 , where f a1 By f a Generate, f b1 and f b2 By f b Generate, f b1 and f b2 Both are of the same nature and can be used interchangeably; then, the kernel generator (KGU) is used to process f. a1 To generate a dynamic convolution kernel; then, filter the features f b1 To obtain the guiding feature f ab The process is shown in the following formula:

[0035] f ab =KGU(f a1 )·f b1

[0036] Here, KGU represents the kernel generation operation. Finally, the final fusion process is shown in the following formula:

[0037] f out =f b ⊕(soft max(f ab )·f b2 )

[0038] Among them, f out This represents the final output feature of the dynamic attention guidance module, ⊕ represents the element-wise addition operation, and softmax represents the softmax function operation.

[0039] Step (4) Constructing the dynamic simulation loss function

[0040] Total loss function L total As shown in the formula below:

[0041] L total =W g ×L g +W b ×L b +W p ×L p

[0042] Among them, L g L b and L p These represent the losses for the Gaussian branch, the main branch, and the complete prediction branch, respectively; L g and L b It is the standard binary cross-entropy loss (L BCE ), L p It is a weighted average loss Weighted binary cross-entropy loss The combination of ; as shown in the following formula:

[0043]

[0044] W g W b and W p These are the change weights for the Gaussian branch, the main branch, and the complete prediction branch, respectively, as shown in the following formula:

[0045]

[0046] Where n represents the total number of complete training cycles, and x represents the current training cycle.

[0047] Step (5) Training of the polyp growth simulation network

[0048] The constructed complete network was trained on the classic polyp segmentation datasets Kvasir and ClinicDB, enabling the network to converge on the polyp segmentation task. The network takes an RGB image as input and outputs a binary polyp prediction map f.P .

[0049] The beneficial effects of this invention are as follows: Based on the natural growth characteristics of polyp images, this invention proposes a novel polyp growth simulation network. This network generates a main image guided by a Gaussian graph and finally fuses all features to form a complete prediction image. This independent branch design avoids the introduction of redundant information, making the training process smoother. To effectively fuse features from different branches, a dynamic attention-guided module is proposed, which can dynamically use features from the previous layer to guide the feature generation of the next layer. These strategies result in better segmentation performance. This invention achieves a more accurate polyp segmentation model, capable of accurately segmenting polyps of various complex shapes and backgrounds. Attached Figure Description

[0050] Figure 1 This is the specific structure of the polyp growth simulation network. The leftmost PVT encoder represents the feature extraction backbone network, with different sizes indicating that the extracted feature map sizes gradually decrease. The middle FEF-P, FEF-B, and FEF-G represent the complete polyp prediction branch, the main body branch, and the Gaussian branch, respectively. Conv represents a combination of convolutional layers, layer normalization, and ReLU activation functions, and DAG represents the dynamic attention guidance module. The three ground truth maps on the right are the complete ground truth map, the main body ground truth map, and the Gaussian ground truth map.

[0051] Figure 2 To illustrate the internal structure of the dynamic attention guidance module, a 1×1 convolution represents a convolutional network with a kernel of 1. ⊕ represents dynamic convolution operation, ⊕ represents pixel addition operation, ⊙ represents element-wise multiplication, softmax represents softmax activation function, and KGU represents convolution kernel generation module.

[0052] Figure 3 Here is an example of truth graph decoupling, where (a) represents a Gaussian truth graph, (b) represents a complete truth graph, and (c) represents a main truth graph. Detailed Implementation

[0053] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0054] A method for segmenting intestinal polyp images to simulate polyp growth, comprising the following steps:

[0055] (1) Constructing a basic feature extraction network

[0056] The feature extraction network uses a PVTv2 network structure as its backbone, such as Figure 1As shown in the left-hand PVT encoder, its input is a 3-channel RGB image. Features f1, f2, f3, and f4, representing 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image size, are extracted from PVTv2 as outputs, with corresponding channel numbers of 64, 128, 320, and 512 channels, respectively.

[0057] (2) Construct a feature extraction and fusion module

[0058] First, the complete truth map is decoupled into a Gaussian truth map and a main truth map, which focus more on location information and main information (excluding edge details), respectively. Figure 3 As shown.

[0059] Encoder features at different levels contain different types of information; higher-level features contain more abstract information, while lower-level features contain richer detailed information. This module has three branches: the Gaussian branch, the main branch, and the complete prediction branch. These branches fuse features from different levels and are supervised by the Gaussian ground truth map, the main ground truth map, and the complete ground truth map. The specifics are as follows:

[0060] Gaussian branches, such as Figure 1 As shown in FEF-G: using only high-level features f4 and f3, the fusion process is as follows:

[0061] f G =F(Cat(F(up(f4))·f3,F(up(f4))))

[0062] Where · represents element-wise multiplication, up(·) represents upsampling, Cat(·) represents concatenation, and F(·) represents a combination of convolution, batch normalization, and ReLU activation functions.

[0063] Main branches, such as Figure 1 As shown in FEF-B: High-level features f4 and f3 and intermediate feature f2 are used. First, the three features are aggregated to form a richer combined feature f′.

[0064] f′=F(F(f4)·f3)·f2

[0065] Then, a Dynamic Attention Guidance Module (DAG) is used to further implement feature fusion:

[0066] f B =DAG(f G ,F(Cat(F(up(f4)),F(up(f3),f′)))

[0067] Complete prediction branch, such as Figure 1 As shown in FEF-P: This branch uses all features f4, f3, f2, and f1.

[0068] First, the enhanced feature f″ is obtained:

[0069] f″=F(F(F(f4)·f3)·f2)·f1

[0070] A Dynamic Attention Guidance Module (DAG) is used to further perform feature fusion, resulting in the final predicted features:

[0071] f P =DAG(f B ,F(Cat(F(f4),F(f3),F(f2),f”)))

[0072] The results of the last three branches are upsampled to the same size as the input image and are supervised by three ground truth graphs respectively.

[0073] (3) Constructing a Dynamic Attention Guidance Module (DAG)

[0074] Ordinary convolution parameters remain unchanged after training, depending on all inputs, lacking flexibility. Dynamic convolution, on the other hand, can adjust its parameter values ​​according to different inputs. The three branches of this invention are closely interconnected, with each branch building upon the next. To achieve more efficient feature fusion, we designed a dynamic convolution attention guidance module based on dynamic convolution and attention mechanisms.

[0075] like Figure 2 As shown, a dynamic attention guidance module is used to fuse guided features f. a and features f with rich semantic information b First, these two features are used to generate f through linear mappings implemented using 1x1 convolutions. a1 f b1 and f b2 Based on f a1 A dynamic convolutional kernel is generated using a kernel generator unit (KGU). Then, the features f are filtered. b1 To obtain the guiding feature f ab The process is shown in the following formula:

[0076] f ab =KGU(f a1 )·f b1

[0077] Here, KGU represents the kernel generation operation. Finally, the final fusion process is implemented as shown in the following formula:

[0078] f out =f b ⊕(softmax(f ab )·f b2 )

[0079] (4) Constructing a dynamic simulation loss function

[0080] The total loss function is shown in the following formula:

[0081] L total =W g ×L g +W b ×L b +W p ×L p

[0082] Among them, L g L b and L p These represent the losses for the Gaussian branch, the main branch, and the complete prediction branch, respectively. L g and L b It is a standard binary cross-entropy loss, L p It is a combination of weighted intersection-union loss and weighted binary cross-entropy loss. The formula is as follows:

[0083]

[0084] W g W b and W p These are the change weights for the three branches, as shown in the following formula:

[0085]

[0086] Here, n represents the total number of training epochs, and x represents the current number of training epochs. This change conforms to the growth pattern of polyps, where initially only the location information of small polyps is available. As the polyps continue to grow, the location information remains relatively stable, while other information becomes more important. Since the changes in weights when subdivided into each training epoch are very small, they will not affect the optimization of the network's backpropagation.

[0087] (5) Training methods

[0088] The entire network was trained on the classic polyp segmentation datasets Kvasir and ClinicDB to achieve convergence for polyp segmentation. In this embodiment, the network optimizer AdamW was used, with a batch size of 16, an exponentially decaying learning rate curve, and an initial learning rate of 0.0001. Images were cropped to a resolution of 352*352.

Claims

1. A method for segmenting intestinal polyp images to simulate polyp growth, characterized in that, A polyp growth simulation network was used for segmentation. This network, employing a deep learning model, primarily consists of a basic feature extraction network, a feature extraction and fusion module, and a dynamic attention guidance module. The basic feature extraction network extracts multi-level features from intestinal polyp images through an encoder. The feature extraction and fusion module further integrates and enhances the raw features extracted by the encoder to output a predicted image. This module includes three branches: a Gaussian branch, a main body branch, and a complete prediction branch. The dynamic attention guidance module more efficiently integrates information from these three branches. The specific steps are as follows: Step (1) Constructing the basic feature extraction network The feature extraction network uses an encoder to extract multi-level feature information from the input image. The feature extraction network uses a PVT network pre-trained on ImageNet as the encoder. Its input is a 3-channel RGB image, and its output is multi-level features f1, f2, f3 and f4, where f1 is a low-level feature, f2 is a mid-level feature, and f3 and f4 are high-level features. Step (2) Constructing a feature extraction and fusion module The feature extraction and fusion module comprises three branches: the Gaussian branch, the main branch, and the complete prediction branch. These branches fuse features at different levels and are supervised by the Gaussian ground truth map, the main ground truth map, and the complete ground truth map, respectively. Details are as follows: Gaussian branch: Using only high-level features f4 and f3, the output feature f of the Gaussian branch is finally obtained. G The fusion process is shown in the following formula: ； in Represents element-wise multiplication. Represents upsampling, This represents a splicing operation. This represents a combination of convolution, batch normalization, and ReLU activation functions. Main branch: Using high-level features f4 and f3 and intermediate feature f2; first, aggregate the three features to form a richer combined feature. : ； Then, a Dynamic Attention Guidance Module (DAG) is used to further achieve feature fusion, obtaining the output feature f of the main branch. B : ； Full prediction branch: using all features f4, f3, f2, and f1; first, the enhanced features are obtained. : ； Then, a Dynamic Attention Guidance Module (DAG) is used to further perform feature fusion to obtain the final predicted features. : ； Finally, the results of the three branches are upsampled to the same size as the input image and supervised by three ground truth maps respectively; The dataset contains images of intestinal polyps and their corresponding complete ground truth maps. The Gaussian ground truth map and the main ground truth map need to be generated decoupled from the complete ground truth map, according to the following rules: Generation of the Gaussian truth map: The complete truth map I is composed of background pixels I bg and foreground pixel I fg It consists of two parts; first, the center point of the foreground pattern is located based on the complete truth map. Specifically, a rectangle is used to completely and closely surround the foreground pattern, and the center point of the rectangle is taken as the center point p of the foreground pattern. c Its coordinates are represented as (x c y c The length of the shorter side of the rectangle is denoted as R. Then, the value of each coordinate point p in the diagram is calculated using the formula and denoted as... Where the coordinates of p are represented as (x, y): ； Finally, the Gaussian ground truth plot is obtained using the following formula. : ； in Represents all coordinate points in the complete truth graph; Generation of the main truth graph: First define I represents the Euclidean distance between two pixels p and q. fg I represents the foreground pixel. bg This represents the background pixels; then, the value of p for each pixel is calculated using the formula. : ； The obtained Normalization is performed to obtain the main truth map. ; Step (3) Constructing the Dynamic Attention Guidance Module (DAG) Fuse guided features f using a dynamic attention guidance module. a and features f with rich semantic information b Specifically, in the network structure, modules are placed in different locations, and their f a and f b Different specific features can be introduced; in the main branch, f a f G f b For f4, f3, and f2; in the complete polyp prediction branch, f a f B f b Let f4, f3, f2, and f1 be the features; first, the two features f a and f b f is generated by implementing linear mapping through 1*1 convolution. a1 f b1 and f b2 , where f a1 By f a Generate, f b1 and f b2 By f b Generate, f b1 and f b2 Both are of the same nature and can be used interchangeably; then, the KGU kernel generation component is used to process f. a1 To generate a dynamic convolution kernel; then, filter the features f b1 To obtain the guiding feature f ab The process is shown in the following formula: ； Where KGU represents the convolution kernel generation operation; finally, the final fusion process is shown in the following formula: ； in, This represents the final output feature of the dynamic attention guidance module. Represents element addition operation, This represents the softmax function operation; Step (4) Constructing the dynamic simulation loss function Total loss function As shown in the formula below: ； Among them, L g L b and L p These represent the losses for the Gaussian branch, the main branch, and the complete prediction branch, respectively; L g and L b It is a standard binary cross-entropy loss. L p It is a weighted average loss Weighted binary cross-entropy loss The combination of ; as shown in the following formula: ； W g W b and W p These are the change weights for the Gaussian branch, the main branch, and the complete prediction branch, respectively, as shown in the following formula: ； Where n represents the total number of complete training cycles, and x represents the current training cycle. Step (5) Training the polyp growth simulation network The constructed complete network was trained on the classic polyp segmentation datasets Kvasir and ClinicDB, enabling the network to converge on the polyp segmentation task. The network takes an RGB image as input and outputs a binary polyp prediction map. .

Images