Bladder tumor image segmentation method and system based on FEBE-Net

Abstract

The invention discloses a bladder tumor image segmentation method and system based on FEBE-Net, which are applied to the technical field of image processing, and adopt a PVTv2 encoder to learn multi-scale feature representation so as to adapt to the change of the size and shape of a bladder tumor; secondly, a feature exploration attention module is provided to fully mine potential local detail information in shallow features extracted by a PVTv2 encoder, noise is eliminated, missing fine-grained details are supplemented for a decoding stage, and the network segmentation performance is improved; meanwhile, a boundary enhancement and refinement module (BER) is provided, the boundary of the bladder tumor is additionally learned through a boundary detection operator to enhance a boundary region in input features of a decoder end, and a final prediction feature map is refined and adjusted; an efficient self-attention calibration decoder module is provided, and global context information and local detail information are gradually and effectively recovered from high-level features and low-level features with the help of boundary clues provided by a BER module.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to a bladder tumor image segmentation method and system based on FEBE-Net. Background Art

[0002] Bladder cancer is one of the top ten most common cancers worldwide. However, the cure rate of early bladder cancer is very high through surgical treatment. Therefore, early detection and removal of bladder tumors are crucial to improving the cure rate of bladder cancer. Currently, cystoscopy is one of the main examination methods for early bladder tumor screening. In clinical practice, in order to reduce the dependence on visual inspection by urologists and improve the detection accuracy of bladder tumors, it is of great significance to develop an automatic, accurate and robust segmentation algorithm for bladder tumors based on cystoscopic images. However, due to the morphological diversity, low contrast, uneven texture and brightness, and blurred boundaries of bladder tumors, the bladder tumor segmentation task remains challenging.

[0003] In recent years, thanks to the powerful feature extraction ability of convolutional neural networks (CNNs), CNNs have achieved excellent performance in medical image segmentation tasks, such as breast ultrasound image segmentation, skin lesion segmentation, multi-organ CT image segmentation, polyp segmentation, liver lesion segmentation, etc. Although CNN-based medical image segmentation methods have achieved initial success, due to the locality of the convolution operation and the limited convolution kernel size, CNNs are usually insufficient in learning global context information, which is crucial for accurately segmenting lesions or organ tissues with large differences in shape and size. To overcome this limitation, some studies have tried to use non-local self-attention mechanisms based on CNN features to simulate global scenes. However, they are only applicable to low-resolution medical images. In high-resolution images, the computational complexity of these modules is too large. Other studies have tried to expand the receptive field of the convolution kernel, but this requires continuous downsampling operations and stacking sufficiently deep convolution layers, which not only leads to the loss of local details and network redundancy, but more importantly, the effective receptive field is still limited to the local area.

[0004] Recently, the emergence of the Vision Transformer (ViT), which does not require any convolution operations and relies only on the attention mechanism, solves the problem of CNN's inability to learn long-range dependencies. Specifically, the Transformer obtains the similarity of all block pairs through the dot product between block vectors, and adaptively extracts and mixes features between all blocks. This gives the Transformer an effective global receptive field and reduces the inductive bias of the model. Therefore, compared with CNN-based methods, the method based on the visual transformer has a stronger ability to extract global information and generalization ability, greatly improving the accuracy of medical image analysis tasks such as recognition, segmentation, and diagnosis. Although the Transformer-based method has achieved remarkable success in medical image segmentation tasks, it still has limitations in capturing fine-grained detail information due to the lack of spatial induced bias when modeling local information, which may result in unclear boundaries of the predicted target area.

[0005] In order to realize the advantages of CNN and Transformer, some recent studies have tried to combine CNN and Transformer, which are mainly divided into serial branch structure and parallel branch structure. The encoder of the serial network stacks CNN layers and Transformer layers in a sequential manner. However, they only use the self-attention mechanism of context modeling at a single scale, while ignoring the cross-scale dependency and consistency. The parallel network uses the Transformer branch to extract global features and the CNN branch to extract local features, and then fuses the features of the two branches. However, the simple and rough fusion ignores the semantic gap between the two multi-level features, and the dual encoder structure running in parallel also greatly increases the amount of calculation.

[0006] At present, most medical image segmentation networks based on deep learning focus on designing some sophisticated modules to enhance the features extracted by CNN or Transformer backbone networks and integrate contextual information, while ignoring the importance of boundary feature information for prediction results, which may lead to inaccurate or discontinuous prediction boundaries, and thus unsatisfactory final prediction results. To solve this problem, some studies have designed special modules to capture boundary feature information, such as reverse attention modules, etc. However, since these methods do not use boundary supervision, the boundary clues they directly explore using foreground (target) supervision may be inaccurate on some medical images, especially those with blurred boundaries. Other studies have attempted to predict boundary areas under boundary supervision. They supplement boundary prediction results into target prediction results, and refine the final segmentation results to a certain extent. However, their boundary prediction tasks and target prediction tasks are independent and do not communicate with each other. These methods can only modify the final target prediction results, but cannot provide boundary clues for the target prediction process.

[0007] Therefore, how to provide a bladder tumor image segmentation method and system that can extract multi-scale global features and fine-grained detail features and generate a bladder tumor prediction mask with clear boundaries is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0008] In view of this, the present invention provides a bladder tumor image segmentation method and system based on FEBE-Net to solve the problems in the background technology.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] On the one hand, the present invention discloses a bladder tumor image segmentation method based on FEBE-Net, comprising:

[0011] Get the RGB image to be segmented;

[0012] The FEBE-Net prediction model was constructed and the PVTv2 encoder was selected as the backbone network to capture the multi-scale global feature information of bladder tumors. The PVTv2 encoder extracted four different levels of pyramid features X i , i = 1, 2, 3, 4; the low-level features X i , i = 1, 2, 3 are sent to the feature exploration and attention module to mine the potential local detail feature information of bladder tumors and filter out irrelevant noise to generate a feature map i=1,2,3; at the same time, the low-level feature X1 and the high-level feature X4 are input into the boundary enhancement and refinement module to generate high-quality boundary clues and will Make four copies Advanced Features X4 and Boundary Clues is sent to the decoder, using boundary clues first Enhance the boundary area in the high-level feature X4, and then restore the global information through a 1×1 convolution unit to obtain the feature map D'4. The feature map D'4 is then subjected to a 1×1 convolution to reduce the number of channels to 1 to generate the initial global prediction map D4. Boundary clues Advanced Features D' i+1 , i = 2, 3, 4 are sent to the proposed efficient self-attention calibration decoding module, using boundary cues i=2,3 pairs of high-level features D' i+1 , i=2,3 is adaptively weighted to gradually guide the network to focus on the noteworthy bladder tumor boundary area. Through the lateral connection of the feature exploration attention module and the efficient self-attention calibration decoding module, the feature map containing rich local detail information is transformed into The high-level feature map D' recovered from the decoder at the previous stage i

[0013] , i=2,3,4 are fused, and the bladder tumor prediction map D is gradually generated by deep decoding i , i=

[0014] 1,2,3; bladder tumor prediction diagram D of the 5-stage output of the decoder i , i = 1, 2, 3, 4, b for deep supervision and the generated boundary clues Perform boundary supervision, design a hybrid supervision strategy, and define the overall loss function L overall To optimize the network;

[0015] The RGB image to be segmented is input into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

[0016] Preferably, in the above-mentioned bladder tumor image segmentation method based on FEBE-Net, the mathematical expression of the FEBE-Net prediction model is:

[0017] X i =PVTv2(X),(i=1,2,3,4);

[0018]

[0019]

[0020] D i =Conv1(D' i ),(i=1,2,3,4);

[0021]

[0022] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the feature exploration and attention module includes 4 branches, each branch performs different convolution operations on low-level features X1, X2, and X3 to capture local information, wherein the first branch uses a 3×3 convolution unit with an expansion rate of 1 to perform feature mining, on this basis, the second branch adds a 3×3 convolution unit with an expansion rate of 3 to perform subsequent feature exploration, and so on, the third branch supplements and explores features through 3 3×3 convolution units with expansion rates of 1, 3, and 5 respectively; the outputs of the first three branches are then connected in series in the channel dimension, and a 1×1 convolution is used to reduce the number of channels. The fourth branch uses self-calibration convolution to capture richer detail features, and then a series operation is performed, and a linear efficient channel attention module (LECA) is used to filter noise. Finally, a 3×3 convolution unit is used to further denoise the features and restore them to the original number of channels.

[0023] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the mathematical expression of the feature exploration attention module is:

[0024] F 11 =DConv1(X i ),(i=1,2,3);

[0025] F 12 =DConv3(DConv1(X i )),(i=1,2,3);

[0026] F 13 =DCon5(DConv3(DConv1(X i ))),(i=1,2,3);

[0027] F1=Conv1(Cat(F 11 ,F 12 ,F 13 ));

[0028] F2=Conv sc (X i ),(i=1,2,3);

[0029] X Fi =BConv3(LECA(Cat(F1,F2)));

[0030] In the above formula, DConvi (i = 1, 3, 5) represents a 3 × 3 convolution kernel with expansion rates of 1, 3, and 5, batchnormalization, and PReLU activation function, Cat (·) represents the concatenation operation in the channel dimension, Conv1 (·) represents a 1 × 1 convolution, and Conv sc stands for self-calibrated convolution, BConv3 represents 3×3 convolution kernel, batch normalization and ReLU activation function.

[0031] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the efficient self-attention calibration decoding module uses a convolutional layer to embed high-level features into overlapping patches, and models local continuity information by combining each patch with information of adjacent patches above, below, left, and right;

[0032] The obtained serialized tokens are passed through three parallel fully connected layers to calculate the query tensor Q, key tensor K and value tensor V respectively; the similarity between the query Q and key tensor K is calculated to obtain the similarity matrix QK T , and then the similarity matrix QK T Divide by Scaling is performed, and finally the weight obtained is Multiply the value tensor V to get the output O SRA ;

[0033] Output O SRA Added to V and then input into the convolution feed-forward module CFF for further decoding operation, the output O is obtained. CFF ;

[0034] Multiply the query tensor Q and the key tensor K to get the fused feature Q r K r , calibrate Q through two branches respectively r K r , and then output O CFF Multiply to strengthen the global feature and get the first global feature O CFF1 and the second global feature O CFF2 ; Among them, the first branch is the channel attention module CAM, and the second branch is the convolutional attention module;

[0035] The enhanced second global feature O CFF2 Upsampling and outputting fine-grained detail features Multiply to supplement the rich local detail features.

[0036] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the boundary enhancement and refinement module includes:

[0037] The Sobel operator is used to calculate the gradients of the low-level feature X1 and the high-level feature X4 in the horizontal and vertical directions respectively to obtain the gradient map in the overall direction.

[0038] The low-level features X1 are respectively combined with the horizontal convolution matrix C x1 And the vertical convolution matrix C y1 Convolution is performed to obtain the horizontal gradient map G x1 And the vertical gradient map G y1 , combined with the horizontal gradient map G x1 And the vertical gradient map G y1 Get the gradient map in the overall direction

[0039] Use Sigmoid activation function to adjust the gradient map Normalize and multiply with the original feature X1 to get the boundary enhancement feature map The same goes for advanced feature X4;

[0040] Enhance the feature map for the boundary and boundary enhancement feature map Perform convolution fusion operation to predict the boundary of bladder tumor

[0041] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the boundary enhancement feature map and boundary enhancement feature map Perform convolution fusion operation to reduce the boundary enhancement feature map using two 1×1 convolution units and boundary enhancement feature map The number of channels is increased to 32; then a bilinear upsampling operation is used to adjust size so that it is The two are of the same size; then they are spliced ​​and fused along the channel dimension, and two consecutive 3×3 convolution units are used to extract boundary information and eliminate irrelevant noise. Finally, a convolution layer with a kernel size of 1 is used to reduce the number of channels to 1 to generate a bladder tumor boundary prediction map.

[0042] Preferably, in the above-mentioned bladder tumor image segmentation method based on FEBE-Net, the bladder tumor prediction image D outputted by the decoder at the five stages is i , i = 1, 2, 3, 4, b for deep supervision and the generated boundary clues Conduct border surveillance;

[0043] The output of the boundary enhancement and refinement module Copy 4 times to get: Boundary clue

[0044] Boundary clues Boundary clues Boundary clues Integrate into input feature X4, input feature D'4, input feature D'3, and use downsampling operation to convert boundary clues into boundary clues Boundary clues Boundary clues The size of is adjusted to be the same as the input feature X4, input feature D'4, and input feature D'3. Then, the downsampled boundary clue map is multiplied with the input feature X4, input feature D'4, and input feature D'3, and the multiplication result is added to the input feature X4, input feature D'4, and input feature D'3 to enhance the boundary areas in these high-level features.

[0045] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, a hybrid supervision strategy is designed to define the overall loss function L overall To optimize the network, including:

[0046] Define the overall loss function L overall :

[0047]

[0048] L segi =L wiou (D i ,GT)+L wbce (D i ,GT)(i=1,2,3,4,b);

[0049]

[0050] In the above formula, The loss function for bladder tumor segmentation in five stages consists of a weighted cross-link loss and a weighted binary cross entropy loss. wiou and wbce constrain the bladder tumor prediction mask from a global and local perspective, respectively; L bnd represents the bladder tumor boundary segmentation loss function, using the Dice loss function, N represents the total number of pixels in the cystoscopic image, and g i and p i denote the ground truth and predicted label of the i-th pixel respectively; ω is the weight factor to balance the segmentation loss and boundary segmentation loss.

[0051] On the other hand, the present invention discloses a bladder tumor image segmentation system based on FEBE-Net, applying the above method, including:

[0052] An acquisition module is used to acquire the RGB image to be segmented;

[0053] Building modules, constructing FEBE-Net prediction model;

[0054] The output module inputs the RGB image to be segmented into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

[0055] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a bladder tumor image segmentation method and system based on FEBE-Net, which can extract multi-scale global features and fine-grained detail features, and generate a bladder tumor prediction mask with clear boundaries. First, due to the large changes in the size and shape of bladder tumors, the PVTv2 encoder is used to learn long-range dependencies. On the one hand, the proposed feature exploration attention module (FEA) is used to explore the local fine-grained detail information in the shallow features extracted by the PVT v2 encoder and eliminate noise, and then supplement it to the high-level features of the corresponding decoder through lateral connections to fill the semantic gaps; on the other hand, an efficient self-attention calibration decoding module (ESCD) is proposed to connect low-level features with high-level features, and use an efficient self-attention mechanism to restore global features, and then merge the calibrated and enhanced global features with fine-grained detail features to restore the local feature details of the tumor. At the same time, a boundary enhancement refinement module (BER) is proposed to capture boundary clues. On the one hand, it is sent to the input features of the first three layers of the decoder for adaptive weighting to highlight the enhanced boundary area; on the other hand, it is added to the prediction output of the last layer of the decoder to achieve complementary fusion and generate a prediction map with clearer boundaries. In addition, we designed a hybrid supervision strategy to achieve dual supervision of the prediction mask and the boundary mask. Finally, in order to verify the segmentation performance of the proposed model, we conducted a large number of experiments on the bladder tumor dataset BtAMU. The segmentation performance indicators of the proposed method are better than other state-of-the-art (SOTA) methods. Among them, on BtAMU, the MDice and MIou indicators of the proposed method reached 0.9126 and 0.8578 respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0057] Figure 1 A schematic diagram of the FEBE-Net prediction model provided by the present invention;

[0058] Figure 2A schematic diagram of a feature exploration and attention module provided by the present invention;

[0059] Figure 3 Schematic diagram of the efficient self-attention calibration decoding module provided by the present invention;

[0060] Figure 4 A schematic diagram of a boundary enhancement and refinement module provided by the present invention;

[0061] Figure 5 The visual qualitative results of the segmentation performance of FEBE-Net on the bladder tumor dataset BTAMU provided by the present invention. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The embodiment of the present invention discloses a Transformer-based network architecture: FEBE-Net, which is used to improve the segmentation accuracy of bladder tumors in cystoscopy. First, the overall architecture of the proposed FEBE-Net is briefly described, and then the proposed feature exploration attention module (FEA), efficient self-attention calibration decoding module (ESCD), and boundary enhancement refinement module (BER) are introduced in detail. Finally, the overall loss function used to train the network is introduced.

[0063] Specifically, the FEBE-Net prediction model is as follows: Figure 1 As shown in the figure, it contains a PVT v2 encoder and the three proposed modules: Feature Exploration Attention Module (FEA), Efficient Self-Attention Calibrated Decoder Module (ESCD), and Boundary Enhancement Refinement Module (BER).

[0064] Specifically, given an input RGB image X∈R H×W×3 ,Since the visual transformer has a stronger ability to capture long-range dependencies and ,feature representations than CNN, the PVT v2 encoder is selected as the ,backbone network to capture the multi-scale global feature information of bladder tumors to adapt to the ,changes in tumor shape and size. The Transformer backbone network PVT v2 extracts 4 different levels of pyramid ,features X i (i=1,2,3,4), where X1, X2, and X3 are low-level features that contain more fine-grained detail information and noise information that is irrelevant to bladder tumor segmentation, and X4 is a high-level semantic feature that contains more feature information that can locate bladder tumors.

[0065] Then, the low-level features Xi (i=1,2,3) is sent to the FEA module to mine the potential local detail feature information of bladder tumor and filter out irrelevant noise to generate feature map At the same time, the low-level features X1 and high-level features X4 are input into the BER module to generate high-quality boundary clues and will Make four copies Next, advanced features X4 and boundary clues Is sent to the decoder, first using Enhance the boundary area in X4, and then restore the global information through a 1×1 convolution unit to obtain the feature map D'4. D'4 is then subjected to a 1×1 convolution to reduce the number of channels to 1 to generate the initial global prediction map D4. Then, Sent to the proposed decoder module ESCD, using boundary cues For high-level features D' i+1 (i=2,3) is adaptively weighted to gradually guide the network to focus on the noteworthy bladder tumor boundary area. Through the lateral connection between the FEA module and the ESCD module, the feature map containing rich local detail information is transformed into The high-level feature map D' recovered from the decoder at the previous stage i (i=2,3,4) are fused and a more accurate bladder tumor prediction map D is gradually generated through deep decoding. i (i=1,2,3); In order to further refine the boundary area in the prediction map, use Fine-tune the prediction graph D1 to achieve Complementing the feature information in D1, we can obtain a bladder tumor prediction map D with clearer boundaries. b .

[0066] During the experiment, the bladder tumor prediction map D output by the decoder at the 5 stages was i (i=1,2,3,4,b) for deep supervision and the generated boundary clues Conduct border monitoring.

[0067] The FEBE-Net prediction model structure is defined as follows:

[0068] X i =PVTv2(X),(i=1,2,3,4);

[0069]

[0070]

[0071] D i =Conv1(D' i),(i=1,2,3,4);

[0072]

[0073] In order to further optimize the above technical solutions, the Feature Exploration Attention Module (FEA) includes:

[0074] Since the low-level features X1, X2, and X3 extracted by the PVT v2 encoder contain rich detail feature information, the captured fine-grained detail information can be supplemented to the high-level features by exploring and paying attention to these features to solve the problem of high-level features losing local detail feature information and small bladder tumors. However, these low-level feature information on the encoder side contains more background noise and irrelevant information, and there is a semantic gap between them and high-level features. Therefore, a feature exploration and attention module (FEA) is proposed to extract local fine-grained detail information in shallow features while filtering noise, and it is horizontally connected between the encoder and decoder to bridge the semantic gap between low-level features and high-level features.

[0075] The structure of the proposed FEA module is as follows Figure 2 As shown in the figure, specifically, the FEA module has 4 branches, each of which performs different convolution operations on low-level features X1, X2, and X3 to capture local information. Among them, the first branch uses a 3×3 convolution unit with an expansion rate of 1 to mine features. On this basis, the second branch adds a 3×3 convolution unit with an expansion rate of 3 to perform subsequent feature exploration. Similarly, the third branch supplements and explores features through 3 3×3 convolution units with expansion rates of 1, 3, and 5 respectively; the outputs of the first three branches are then connected in series in the channel dimension to expand the receptive field, which can more effectively identify large bladder tumors. Next, a 1×1 convolution is used to reduce the number of channels and reduce computing resources. Different from the first three branches, the fourth branch uses self-calibration convolution to capture richer detail features to enhance the recognition of small bladder tumors, and then performs a series operation, which can help enhance features and capture fine-grained detail features. Subsequently, the linear efficient channel attention module (LECA) is used to filter noise, purify feature information, and force the network to focus on feature information closely related to bladder tumor segmentation. Finally, a 3×3 convolution unit is used to further denoise the features and restore them to the original number of channels. In this way, the FEA module can mine potential local detail feature information of bladder tumors and enhance feature representation. The process can be described as follows:

[0076] F 11 =DConv1(X i ),(i=1,2,3);

[0077] F 12=DConv3(DConv1(X i )),(i=1,2,3);

[0078] F 13 =DCon5(DConv3(DConv1(X i ))),(i=1,2,3);

[0079] F1=Conv1(Cat(F 11 ,F 12 ,F 13 ));

[0080] F2=Conv sc (X i ),(i=1,2,3);

[0081] X Fi =BConv3(LECA(Cat(F1,F2)));

[0082] In the above formula, DConvi (i = 1, 3, 5) represents a 3 × 3 convolution kernel with expansion rates of 1, 3, and 5, batchnormalization, and PReLU activation function, Cat (·) represents the concatenation operation in the channel dimension, Conv1 (·) represents a 1 × 1 convolution, and Conv sc stands for self-calibrated convolution, BConv3 represents 3×3 convolution kernel, batch normalization and ReLU activation function.

[0083] In order to further optimize the above technical solution, the efficient self-attention calibration decoding module (ESCD) includes:

[0084] Since high-level features usually contain more semantic information that can identify bladder tumor areas, however, it may lose some bladder tumor detail information or small bladder tumors. Therefore, an efficient self-attention calibration decoding module (ESCD) is proposed to fuse the fine-grained detail information in the low-level features captured by the FEA module with the global semantic information in the high-level features, effectively restoring the local feature details of the tumor while achieving accurate bladder tumor localization.

[0085] The structure of the proposed ESCD module is as follows: Figure 3As shown in the figure, first, a convolutional layer with a stride of 4 and a kernel size of 4 is used to embed high-level features into overlapping patches. Local continuity information is modeled by combining the information of each block with the adjacent blocks above, below, left, and right. Then, the obtained serialized tokens are passed through three parallel fully connected layers to calculate the query tensor Q, key tensor K, and value tensor V respectively. Next, the obtained serialized tokens are passed through three parallel fully connected layers to calculate the query tensor Q, key tensor K, and value tensor V respectively; the similarity between the query Q and the key tensor K is calculated to obtain the similarity matrix QK T , and then the similarity matrix QK T Divide by Scaling is performed, and finally the weight obtained is Multiply the value tensor V to get the output O SRA ; This process can be described by the formula:

[0086]

[0087] In the formula, Represents the dimension of feature embedding. SRA Added to V and then input into the convolution feed-forward module CFF for further decoding operation, the output is O CFF , this process can be described by formula (16-17):

[0088] O CFF =O SRA +V+CFF(O SRA +V);

[0089] CFF(x)=fc2(R(DWConv(fc1(x))));

[0090] Where x is the input feature, fc1(·) and fc2(·) are linear fully connected mapping layers with a dilation rate of 4, DWConv(·) is a deep convolutional layer that is used to model position information and reduce the amount of computation, and R(·) represents the ReLU activation function.

[0091] Since the query tensor Q and key tensor K carry inherent contextual statistical information, the query tensor Q and key tensor K are multiplied to obtain the fused feature Q r K r , calibrate Q through two branches respectively r K r , filter out more valuable context key information, and then sequentially CFFMultiply to strengthen global features and eliminate noise. The first branch is the channel attention module CAM, and the second branch is the convolutional attention module, which consists of a 3×3 convolution unit and a Sigmoid function. Due to the similarity of K and V, these two branches can weaken irrelevant contextual information from the channel dimension and spatial dimension respectively. This process can be described by the formula:

[0092] Q r K r =r(Q)×r(K)

[0093] O CFF1 =CAM(Q r K r )×O CFF ;

[0094] O CFF2 =S(BConv3(Q r K r ))×O CFF1 ;

[0095] In the above formula, r(·) represents the feature reshape operation, BConv3 represents the 3×3 convolution kernel, batch normalization and ReLU activation function, and S(·) represents the Sigmoid activation function.

[0096] Then, the enhanced global feature O CFF2 Upsampling, fine-grained detail features output by the FEA module Multiplying together to supplement the rich local detail features helps the decoder module restore fine-grained details when predicting bladder tumor regions. This process can be described by the formula:

[0097]

[0098] In order to further optimize the above technical solution, the boundary enhancement and refinement module (BER) includes:

[0099] Since boundary feature information can provide useful fine-grained features, it can help enhance the learning ability of medical image semantic segmentation. Therefore, a boundary enhancement and refinement module (BER) is proposed, which directly uses the first and fourth layer features extracted by the encoder to generate high-quality boundary clues, and then integrates them into the bladder tumor prediction task, aiming to enhance the boundary area in the high-level features on the decoder side and fine-tune the final prediction map, so that the network generates a prediction mask with more accurate positioning and clearer boundaries.

[0100] The structure of the proposed boundary enhancement and refinement module is as follows: Figure 4As shown, the boundary feature information in the low-level feature X1 and the high-level feature X4 is extracted, and the boundary irrelevant information is filtered out.

[0101] First, the Sobel operator is used to calculate the gradients of the low-level feature X1 and the high-level feature X4 in the horizontal and vertical directions respectively to obtain the gradient map in the overall direction. Specifically, taking the low-level feature X1 as an example, the horizontal and vertical convolution C x1 and C y1 They are defined as:

[0102]

[0103] Then, X1 and C x1 and C y1 Convolution is performed to obtain the gradient map G in the horizontal and vertical directions x1 and G y1 , combined with G x1 and G y1 Get the gradient map in the overall direction

[0104]

[0105] Next, use the Sigmoid activation function to Normalize and multiply with the original feature X1 to get the boundary enhancement feature map For the advanced feature X4, the same logic can be obtained:

[0106]

[0107]

[0108] In the formula, S(·) represents the Sigmoid activation function.

[0109] Then, the boundary enhancement feature map and Perform a series of convolution fusion operations. Specifically, use two 1×1 convolution units to reduce and The number of channels is increased to 32, which further extracts features while reducing the consumption of computing resources; then a bilinear upsampling operation is used to adjust size so that it is The two are of the same size; then they are spliced ​​and fused along the channel dimension, and two consecutive 3×3 convolution units are used to further extract boundary information and eliminate irrelevant noise. Finally, a convolution layer with a kernel size of 1 is used to reduce the number of channels to 1 to generate a bladder tumor boundary prediction map. This process can be described by the following formula:

[0110]

[0111]

[0112] In the above formula, BConv1 represents a 1×1 convolution kernel, batch normalization, and ReLU activation function. Cat(·) represents the connection operation in the channel dimension, BConv3 represents a 3×3 convolution kernel, batch normalization, and ReLU activation function. Conv1(·) represents a 1×1 convolution.

[0113] At the same time, the same Sobel operation is applied to each bladder tumor mask in the ground truth image (GT) to generate the corresponding tumor boundary mask to obtain the ground truth boundary image (GTB). Boundary supervision can eliminate the edge feature information inside the bladder tumor and obtain high-quality bladder tumor boundary clues.

[0114] The output of the BER module Make 4 copies and you get: Since the input features X4, D4, and D3 of the first three layers of the decoder contain more advanced semantic concepts, however, there is no clear boundary between the foreground and background, Integrate into high-level features X4, D4, D3. Specifically, use downsampling operations to transform boundary clues The size of is adjusted to the same as X4, D4, D3, and then the downsampled boundary clue map is multiplied with X4, D4, D3, and the multiplication result is added to X4, D4, D3 to enhance the boundary area in these high-level features, aiming to gradually guide the network to focus on the noteworthy boundary area, thereby generating a more accurate bladder tumor prediction map. In addition, The final prediction feature map is fine-tuned to achieve information complementarity and further refine the boundary area of ​​the bladder tumor prediction map.

[0115] like Figure 1 As shown in the figure, the proposed FEBE-Net requires a ground truth map (GT) and a ground truth boundary map (GTB) to predict the mask D of the bladder tumor respectively. i (i=1,2,3,4,b) and tumor boundary prediction mask Therefore, a hybrid supervision strategy is designed to define an overall loss function L overall To optimize the network:

[0116]

[0117] In the above formula, Represents the loss function of the 5-stage bladder tumor segmentation, which consists of the weighted cross-link loss (wiou) and the weighted binary cross entropy loss (wbce), which constrain the bladder tumor prediction mask from the global and local perspectives, respectively. bnd represents the bladder tumor boundary segmentation loss function, which uses the Dice loss function to alleviate the negative impact of the area imbalance between foreground and background pixels in boundary segmentation. N represents the total number of pixels in the cystoscopic image, g i and p i denote the ground truth and predicted label of the i-th pixel respectively. ω is the weight factor to balance the segmentation loss and boundary segmentation loss, which is set to 5 in this paper.

[0118] First, we introduce the dataset, evaluation metrics, and implementation details. The results of the proposed FEBE-Net are compared with the state-of-the-art (SOTA) methods to demonstrate the superiority of the proposed FEBE-Net. Ablation experiments are performed to demonstrate the effectiveness of the three proposed new modules.

[0119] Bladder tumor dataset BtAMU:

[0120] BtAMU consists of 1948 bladder tumor images and their corresponding ground truth labels extracted from 110 4K cystoscopy videos provided by the Urology Department of the First Affiliated Hospital of Anhui Medical University. The GT is manually annotated by professional urology experts, and the image resolution is 1920×1080. When conducting bladder tumor segmentation experiments, 1562 images from the BTAMU dataset were randomly selected at a ratio of 8:2 as the training dataset, and the remaining 386 images were used as the test dataset.

[0121] In the bladder tumor segmentation experiment, the average Dice coefficient (mDice), average intersection over union (mIoU), mean absolute error (MAE), accuracy (Accuracy), weighted and Hausdorff distance (HD) as evaluation indicators for quantitative analysis; in the polyp segmentation task, mDice, mIoU and HD are used as evaluation indicators. mDice and mIoU are similarity metrics at the regional level, indicating the degree of consistency between the model prediction results and the actual results; MAE is a pixel-level evaluation indicator, indicating the mean absolute error between the algorithm prediction value and the actual value; Accuracy indicates the proportion of pixels correctly predicted by the model to the actual pixels; It is an indicator that comprehensively considers Precision and Recall. It weighs Precision and Recall through a coefficient β. HD is used to measure the accuracy of boundary segmentation. Among them, mDice, mIoU, Accuracy and The higher the value of the indicator, the better the prediction effect of the algorithm, and the lower the value of MAE and HD indicators, the better the segmentation effect of the model. They are defined as follows:

[0122]

[0123] In the above formula, TP stands for true positive, TN stands for true negative, FP stands for false positive, and FN stands for false positive; N stands for the number of test images. In the formula, P stands for the predicted image, Y stands for the ground truth, W and H are the width and height of the image respectively; in the formula, β is a parameter, Precision ω and Recall ω They represent weighted precision and weighted recall respectively, and β in this paper is taken as 1. In the formula, A and B represent the real image and the predicted image, and ||ab|| represents the distance function, such as the Euclidean distance function.

[0124] The proposed FEBE-Net is implemented based on the PyTorch framework, and the graphics card memory is 24GB and the graphics card configuration is NVIDIA GeForce RTX 3090 to accelerate the calculation. The optimizer is set to AdamW and the initial learning rate is set to 5×10 -5 The weight decay coefficient (decayrate) is set to 0.1, the batch size is set to 8, and a total of 100 epochs are trained. In order to accelerate the convergence of the network, the learning rate is decreased with a decay rate of 0.5 after the 15th and 30th epochs. In addition, the input image is resized to 352×352, and a multi-scale {0.75, 1.0, 1.25} training strategy with a gradient clipping limit of 0.5 is used.

[0125] In order to verify the effectiveness of the proposed FEBE-Net for bladder tumor segmentation, the quantitative results of the proposed FEBE-Net are compared with those of UNet, PraNet, CaraNet, HarDNet-MSEG, DCRNet, MSRAformer, HSNet, and TGDAUNet, and the mDice, mIoU, MAE, Accuracy, and HD 6 medical image segmentation evaluation indicators are used to quantitatively evaluate the segmentation performance of all models on the bladder tumor dataset BtAMU. Table 1 lists the quantitative results of different algorithms on the bladder tumor dataset BtAMU. It can be seen from Table 1 that compared with other models, the proposed FEBE-Net achieves the most advanced segmentation performance on the BtAMU dataset. In terms of mDice and mIoU indicators, FEBE-Net is 1.4% and 1.63% higher than the second-best MSRAformer, respectively, which indicates that FEBE-Net can better distinguish bladder tumors from normal tissue background. Similarly, FEBE-Net's Accuracy and It is also 0.16% and 0.52% higher than the second-best MSRAformer. In addition, FEBE-Net is 0.22% and 0.3454 lower than the second-best HSNet in MAE and HD indicators. The HD quantitative results show that FEBE-Net improves the accuracy of bladder tumor boundary segmentation and reduces boundary segmentation error. These results verify the effectiveness and superiority of the proposed FEBE-Net for bladder tumor segmentation in cystoscopic images.

[0126] Table 1

[0127]

[0128] Quantitative results of the FEBE-Net-based prediction model on the BtAMU dataset compared with other state-of-the-art (SOTA) methods. ↑ indicates higher is better, ↓ indicates lower is better, and bold indicates the best performance.

[0129] In order to more clearly and intuitively demonstrate the effectiveness of the proposed FEBE-Net, Figure 5 The visualization results of FEBE-Net on the bladder tumor dataset BtAMU are given. From top to bottom, they are the original bladder tumor image, the ground truth image (GT), the bladder tumor mask predicted by FEBE-Net, and the visualization result of the comparison between the predicted image of FEBE-Net and GT. In the fourth row, white represents the area predicted by FEBE-Net and presented in GT (TP, correct prediction, the predicted result is positive, and the actual is positive). Blue represents the area predicted by BT-PVT but does not exist in GT (FP, wrong prediction, the predicted result is positive, and the actual is negative), and the red area is a false detection. Red represents the area that exists in GT but is not predicted by BT-PVT (FN, wrong prediction, the predicted result is negative, and the actual is positive), which is a missed detection. From Figure 5 It can be seen that FEBE-Net can effectively solve the challenges brought by the different shapes and sizes of bladder tumors and uneven brightness.

[0130] Specific Figure 5 Visual qualitative results of the segmentation performance of the proposed FEBE-Net on the bladder tumor dataset BTAMU. The first row shows images of bladder tumors of different shapes and sizes under cystoscopic observation. The second row shows the ground truth image (GT) of the bladder tumor dataset BTAMU. The third row shows the bladder tumor mask predicted by FEBE-Net. The fourth row shows the visualization result of the comparison between the prediction map of FEBE-Net and GT.

[0131] In the proposed FEBE-Net, three new modules (FEA, ESCD, BER) are proposed to improve the bladder tumor segmentation performance. In order to verify the effectiveness of each module, an ablation experiment is performed on the bladder tumor dataset BT-AMU to explore the impact of each module on the bladder tumor segmentation performance.

[0132] Specifically, the baseline is PVT v2; for the verification of the effectiveness of the FEA module, it is directly removed; for the ESCD module, it is replaced with a 1×1 convolution unit and a two-fold upsampling operation for comparison; for the BER module, it is directly removed to verify its effectiveness. The ablation experiments of the above FEA, ESCD and BER modules are marked as "w / oFEA", "w / o ESCD" and "w / o BER" ​​respectively. The experimental results are shown in Table 2. It can be seen from Table 2 that replacing or removing any of the proposed modules will lead to a significant decrease in the bladder tumor segmentation performance, which proves the effectiveness of the proposed modules; the segmentation performance of "w / o FEA", "w / o ESCD" and "w / o BER" ​​are all higher than the baseline, reflecting the effectiveness of the cooperation between the modules.

[0133] Table 2

[0134]

[0135] Quantitative results of FEBE-Net's ablation experiment on the bladder tumor dataset BTAMU. ↑ means the higher the better, ↓ means the lower the better, and bold black indicates the best performance.

[0136] In summary, the feature exploration attention module (FEA) mines the potential detail information in low-level features through different convolution operations, solves the problem of high-level features losing local detail feature information and small bladder tumors, and uses the LECA mechanism and lateral connection method to help bridge the semantic gap between features and improve the segmentation accuracy of bladder tumors.

[0137] The efficient self-attention calibrated decoding module (ESCD) effectively restores the global features through the efficient self-attention module, and fuses the local detail features with the calibrated and enhanced global features, effectively restoring the detail features, enhancing the feature representation ability of the model, and solving the problem that Transformer cannot accurately represent local details.

[0138] The boundary enhancement and refinement module (BER) effectively extracts bladder tumor boundaries through boundary detection operators, which not only provides favorable boundary clues for the decoder to predict bladder tumor areas, but also refines and adjusts the final bladder tumor prediction map. Different from existing methods, BER directly uses boundary supervision to explore boundary clues, and communicates the bladder tumor boundary extraction task and the tumor prediction task, thereby improving the learning ability of bladder tumor semantic segmentation.

[0139] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0140] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A bladder tumor image segmentation method based on FEBE-Net, characterized in that: include: Get the RGB image to be segmented; The FEBE-Net prediction model was constructed and the PVTv2 encoder was selected as the backbone network to capture the multi-scale global feature information of bladder tumors. The PVTv2 encoder extracted four different levels of pyramid features X i , i = 1, 2, 3, 4; the low-level features X i , i = 1, 2, 3 are sent to the feature exploration and attention module to mine the potential local detail feature information of bladder tumors and filter out irrelevant noise to generate a feature map i=1,2,3; at the same time, the low-level feature X1 and the high-level feature X4 are input into the boundary enhancement and refinement module to generate high-quality boundary clues and will Make four copies Advanced Features X4 and Boundary Clues is sent to the decoder, using boundary clues first Enhance the boundary area in the high-level feature X4, and then restore the global information through a 1×1 convolution unit to obtain the feature map D'4. The feature map D'4 is then subjected to a 1×1 convolution to reduce the number of channels to 1 to generate the initial global prediction map D4. Boundary clues Advanced Features D' i+1 , i = 2, 3, 4 are sent to the proposed efficient self-attention calibration decoding module, using boundary cues i=2,3 pairs of high-level features D' i+1 , i=2,3 is adaptively weighted to gradually guide the network to focus on the noteworthy bladder tumor boundary area. Through the lateral connection of the feature exploration attention module and the efficient self-attention calibration decoding module, the feature map containing rich local detail information is transformed into The high-level feature map D' recovered from the decoder at the previous stage i , i=2,3,4 are fused, and the bladder tumor prediction map D is gradually generated by deep decoding i ,i=1,2,3;The bladder tumor prediction map D output by the 5 stages of the decoder i , i = 1, 2, 3, 4, b for deep supervision and the generated boundary clues Perform boundary supervision, design a hybrid supervision strategy, and define the overall loss function L overall To optimize the network; The RGB image to be segmented is input into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

2. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that: The mathematical expression of the FEBE-Net prediction model is: X i =PVTv2(X),(i=1,2,3,4); D i =Conv1(D′ i ),(i=1,2,3,4); 3. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that: The feature exploration and attention module includes 4 branches, each of which performs different convolution operations on low-level features X1, X2, and X3 to capture local information. The first branch uses a 3×3 convolution unit with an expansion rate of 1 to perform feature mining. On this basis, the second branch adds a 3×3 convolution unit with an expansion rate of 3 to perform subsequent feature exploration. Similarly, the third branch supplements and explores features through 3 3×3 convolution units with expansion rates of 1, 3, and 5 respectively; the outputs of the first three branches are then connected in series in the channel dimension, and a 1×1 convolution is used to reduce the number of channels. The fourth branch uses self-calibrated convolution to capture richer detail features, and then connects them in series, using a linear and efficient channel attention module to filter noise. Finally, a 3×3 convolution unit is used to further denoise the features and restore them to the original number of channels.

4. The bladder tumor image segmentation method based on FEBE-Net according to claim 3, characterized in that: The mathematical expression of the feature exploration attention module is: F 11 =DConv1(X i ),(i=1,2,3); F 12 =DConv3(DConv1(X i )),(i=1,2,3); F 13 =DCon5(DConv3(DConv1(X i ))),(i=1,2,3); F1=Conv1(Cat(F 11 ,F 12 ,F 13 )); F2=Conv sc (X i ),(i=1,2,3); X Fi =BConv3(LECA(Cat(F1,F2))); In the above formula, DConvi (i = 1, 3, 5) represents a 3 × 3 convolution kernel with expansion rates of 1, 3, and 5, batchnormalization, and PReLU activation function, Cat (·) represents the concatenation operation in the channel dimension, Conv1 (·) represents a 1 × 1 convolution, and Conv sc stands for self-calibrated convolution, BConv3 represents 3×3 convolution kernel, batch normalization and ReLU activation function.

5. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that: The efficient self-attention calibration decoding module uses convolutional layers to embed high-level features into overlapping patches and models local continuity information by combining the information of each patch with its upper, lower, left, and right neighboring patches; The obtained serialized tokens are passed through three parallel fully connected layers to calculate the query tensor Q, key tensor K and value tensor V respectively; the similarity between the query Q and key tensor K is calculated to obtain the similarity matrix QK T , and then the similarity matrix QK T Divide by Scaling is performed, and finally the weight obtained is Multiply the value tensor V to get the output O SRA ; Output O SRA Added to V and then input into the convolution feed-forward module CFF for further decoding operation, the output O is obtained. CFF ; Multiply the query tensor Q and the key tensor K to get the fused feature Q r K r , calibrate Q through two branches respectively r K r , and then output O CFF Multiply to strengthen the global feature and get the first global feature O CFF1 and the second global feature O CFF2 ; Among them, the first branch is the channel attention module CAM, and the second branch is the convolutional attention module; The enhanced second global feature O CFF2 Upsampling and outputting fine-grained detail features Multiply to supplement the rich local detail features.

6. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that: The boundary enhancement and refinement module includes: The Sobel operator is used to calculate the gradients of the low-level feature X1 and the high-level feature X4 in the horizontal and vertical directions respectively to obtain the gradient map in the overall direction. The low-level features X1 are respectively combined with the horizontal convolution matrix C x1 And the vertical convolution matrix C y1 Convolution is performed to obtain the horizontal gradient map G x1 And the vertical gradient map G y1 , combined with the horizontal gradient map G x1 And the vertical gradient map G y1 Get the gradient map in the overall direction Use Sigmoid activation function to adjust the gradient map Normalize and multiply with the original feature X1 to get the boundary enhancement feature map The same goes for advanced feature X4; Enhance the feature map for the boundary and boundary enhancement feature map Perform convolution fusion operation to predict the boundary of bladder tumor 7. The bladder tumor image segmentation method based on FEBE-Net according to claim 6, characterized in that: Enhance the feature map for the boundary and boundary enhancement feature map Perform convolution fusion operation to reduce the boundary enhancement feature map using two 1×1 convolution units and boundary enhancement feature map The number of channels is increased to 32; then a bilinear upsampling operation is used to adjust size so that it is The two are of the same size; then they are spliced ​​and fused along the channel dimension, and two consecutive 3×3 convolution units are used to extract boundary information and eliminate irrelevant noise. Finally, a convolution layer with a kernel size of 1 is used to reduce the number of channels to 1 to generate a bladder tumor boundary prediction map.

8. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that: Bladder tumor prediction map D output by the 5 stages of the decoder i , i = 1, 2, 3, 4, b for deep supervision and the generated boundary clues Conduct border surveillance; The output of the boundary enhancement and refinement module Copy 4 times to get: Boundary clue Boundary clues Boundary clues Boundary clues Integrate into input feature X4, input feature D'4, input feature D'3, and use downsampling operation to convert boundary clues into boundary clues. Boundary clues Boundary clues The size of is adjusted to be the same as the input feature X4, input feature D'4, input feature D'3, and then the downsampled boundary clue map is multiplied with the input feature X4, input feature D'4, input feature D'3, and the multiplication result is added to the input feature X4, input feature D'4, input feature D'3 to enhance the boundary area in the input feature X4, input feature D'4, input feature D'3.

9. The bladder tumor image segmentation method based on FEBE-Net according to claim 6, characterized in that: Design a hybrid supervision strategy and define the overall loss function L overall To optimize the network, including: Define the overall loss function L overall : In the above formula, The loss function for bladder tumor segmentation in five stages consists of a weighted cross-link loss and a weighted binary cross entropy loss. wiou and wbce constrain the bladder tumor prediction mask from a global and local perspective, respectively; L bnd represents the bladder tumor boundary segmentation loss function, using the Dice loss function, N represents the total number of pixels in the cystoscopic image, and g i and p i denote the ground truth and predicted label of the i-th pixel respectively; ω is the weight factor to balance the segmentation loss and boundary segmentation loss.

10. A bladder tumor image segmentation system based on FEBE-Net, applying the bladder tumor image segmentation method based on FEBE-Net according to any one of claims 1 to 9, characterized in that: include: An acquisition module is used to acquire the RGB image to be segmented; Building modules, constructing FEBE-Net prediction model; The output module inputs the RGB image to be segmented into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

Images