Bladder tumor image segmentation method and system based on febe-net

Abstract

This invention discloses a bladder tumor image segmentation method and system based on FEBE-Net, applied in the field of image processing technology. The invention employs a PVTv2 encoder to learn multi-scale feature representations to adapt to variations in bladder tumor size and shape. Secondly, a feature exploration attention module is proposed to fully mine potential local details in the shallow features extracted by the PVTv2 encoder and eliminate noise, supplementing missing fine-grained details for the decoding stage and improving network segmentation performance. Simultaneously, a boundary enhancement and refinement (BER) module is proposed, which uses a boundary detection operator to additionally learn the boundaries of the bladder tumor to enhance the boundary regions in the input features of the decoder and refine the final predicted feature map. Finally, an efficient self-attention calibration decoder module is proposed, which, with the help of boundary cues provided by the BER module, gradually and effectively recovers global contextual information and local detail information from high-level and low-level features.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a bladder tumor image segmentation method and system based on FEBE-Net. Background Technology

[0002] Bladder cancer is one of the ten most common cancers worldwide. However, early-stage bladder cancer has a high cure rate with surgical treatment; therefore, early detection and removal of bladder tumors are crucial for improving the cure rate. Currently, cystoscopy is one of the main methods for early bladder tumor screening. In clinical practice, developing an automatic, accurate, and robust bladder tumor segmentation algorithm based on cystoscopic images is of great significance in order to reduce reliance on visual examination by urologists and improve the accuracy of bladder tumor detection. However, due to the morphological diversity, low contrast, uneven texture and brightness, and blurred boundaries of bladder tumors, bladder tumor segmentation remains challenging.

[0003] In recent years, thanks to the powerful feature extraction capabilities of Convolutional Neural Networks (CNNs), CNNs have achieved excellent performance in medical image segmentation tasks, such as breast ultrasound image segmentation, skin lesion segmentation, multi-organ CT image segmentation, polyp segmentation, and liver lesion segmentation. Although CNN-based medical image segmentation methods have achieved initial success, due to the locality of convolutional operations and the limited size of the convolutional kernel, CNNs are generally insufficient in learning global contextual information, which is crucial for accurately segmenting lesions or organ tissues with significant differences in shape and size. To overcome this limitation, some studies have attempted to use nonlocal self-attention mechanisms based on CNN features to simulate global scenes; however, these are only applicable to low-resolution medical images, and the computational complexity of these modules is too high in high-resolution images. Other studies have attempted to expand the receptive field of the convolutional kernel, but this requires continuous downsampling operations and stacking sufficiently deep convolutional layers, which not only leads to the loss of local details and network redundancy, but more importantly, the effective receptive field remains limited to local regions.

[0004] Recently, the emergence of the VisionTransformer (ViT), which relies solely on attention mechanisms without any convolutional operations, has addressed the shortcomings of CNNs in learning long-range dependencies. Specifically, the Transformer obtains the similarity of all block pairs through the dot product between block vectors, adaptively extracting and blending features across all blocks. This gives the Transformer an effective global receptive field and reduces the model's inductive bias. Therefore, compared to CNN-based methods, VisionTransformer-based methods have a stronger ability to extract global information and generalize, significantly improving the accuracy of medical image analysis tasks such as recognition, segmentation, and diagnosis. Although Transformer-based methods have achieved significant success in medical image segmentation, they still have limitations in capturing fine-grained details due to the lack of spatially induced bias when modeling local information, which may lead to unclear boundaries of predicted target regions.

[0005] To leverage the advantages of CNNs and Transformers, recent research has attempted to combine them, primarily through sequential and parallel branching structures. Sequential networks stack CNN and Transformer layers sequentially; however, they only utilize context-based self-attention at a single scale, neglecting cross-scale dependencies and consistency. Parallel networks extract global features from Transformer branches and local features from CNN branches, then fuse the features from both branches. However, this simplistic and coarse fusion ignores the semantic differences between the two multi-level features, and the parallel dual-encoder structure significantly increases computational cost.

[0006] Currently, most deep learning-based medical image segmentation networks focus on designing sophisticated modules to enhance features extracted by CNN or Transformer backbone networks and integrate contextual information, while neglecting the importance of boundary features for prediction results. This can lead to inaccurate or discontinuous prediction boundaries, resulting in unsatisfactory final predictions. To address this issue, some studies have designed specialized modules to capture boundary features, such as inverse attention modules. However, since these methods do not employ boundary supervision, their direct use of foreground (target) supervision to explore boundary cues may be inaccurate on some medical images, especially those with blurred boundaries. Other studies have attempted to predict boundary regions under boundary supervision, supplementing the target prediction results with boundary predictions and refining the final segmentation results to some extent. However, their boundary prediction and target prediction tasks are independent and do not interact. These methods can only modify the final target prediction result, but cannot provide boundary cues for the target prediction process.

[0007] Therefore, how to provide a bladder tumor image segmentation method and system that can extract multi-scale global features and fine-grained detail features and generate a bladder tumor prediction mask with clear boundaries is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a bladder tumor image segmentation method and system based on FEBE-Net to solve the problems in the background art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] On one hand, this invention discloses a bladder tumor image segmentation method based on FEBE-Net, including:

[0011] Obtain the RGB image to be segmented;

[0012] The FEBE-Net prediction model was constructed, and the PVTv2 encoder was selected as the backbone network to capture multi-scale global feature information of bladder tumors. The PVTv2 encoder extracted pyramid features X at four different levels. i , i = 1, 2, 3, 4; The low-level feature X i The numbers i=1,2,3 are sent to the feature exploration and focus module to mine potential local detailed features of bladder tumors and filter out irrelevant noise, generating feature maps. i = 1, 2, 3; simultaneously, low-level feature X1 and high-level feature X4 are input into the boundary enhancement and refinement module to generate high-quality boundary cues. And Copy four times to get Advanced features X4 and boundary cues The data is sent to the decoder and first uses boundary clues. Enhance the boundary regions in high-level feature X4, then recover global information through a 1×1 convolutional unit to obtain feature map D'4. Feature map D'4 is then passed through a 1×1 convolution to reduce the number of channels to 1 to generate the initial global prediction map D4. Boundary Clues Advanced Feature D' i+1 The values ​​i = 2, 3, and 4 are sent to the proposed efficient self-attention calibration decoding module, utilizing boundary cues. i = 2, 3 pairs of high-level features D' i+1 Adaptive weighting is applied to i=2 and i=3 to gradually guide the network to focus on noteworthy bladder tumor boundary regions. Through the lateral connection between the feature exploration and attention module and the efficient self-attention calibration decoding module, feature maps containing rich local details are generated. The high-level feature map D' recovered from the decoder in the previous stage i The bladder tumor prediction map D is generated step by step through deep decoding, where i = 2, 3, and 4 are fused. i , i = 1, 2, 3; bladder tumor prediction map D output from the 5 stages of the decoder. i Deep supervision is performed on i = 1, 2, 3, 4, b, and the generated boundary clues are analyzed. Boundary supervision is performed, and a hybrid supervision strategy is designed to define the overall loss function L. overall To optimize the network;

[0013] The RGB image to be segmented is input into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

[0014] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the mathematical expression of the FEBE-Net prediction model is:

[0015] X i =PVTv2(X), (i = 1, 2, 3, 4);

[0016]

[0017]

[0018] D i =Conv1(D' i ), (i = 1, 2, 3, 4);

[0019]

[0020] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the feature exploration and attention module includes four branches. Each branch performs different convolution operations on low-level features X1, X2, and X3 to capture local information. Specifically, the first branch uses a 3×3 convolutional unit with a scaling factor of 1 for feature mining. Based on this, the second branch adds another 3×3 convolutional unit with a scaling factor of 3 for subsequent feature exploration, and so on. The third branch supplements and explores features using three 3×3 convolutional units with scaling factors of 1, 3, and 5, respectively. The outputs of the first three branches are then concatenated along the channel dimension, using a 1×1 convolution to reduce the number of channels. The fourth branch uses a self-calibrating convolution to capture richer detailed features, and then performs a concatenation operation. The Linear Efficient Channel Attention Module (LECA) is used to filter noise. Finally, a 3×3 convolutional unit is used to further denoise the features and restore the original number of channels.

[0021] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the mathematical expression for the feature exploration focus module is:

[0022] F 11 =DConv1(X i ), (i = 1, 2, 3);

[0023] F 12 =DConv3(DConv1(X i ),(i=1,2,3);

[0024] F 13 =DCon5(DConv3(DConv1(X) i ))),(i=1,2,3);

[0025] F1 = Conv1(Cat(F 11 ,F 12 ,F 13 ));

[0026] F2 = Conv sc (X i ), (i = 1, 2, 3);

[0027] X Fi =BConv3(LECA(Cat(F1,F2)));

[0028] In the above formula, DConvi(i=1,3,5) represents a 3×3 convolution kernel with expansion ratios of 1, 3, and 5, respectively, batch normalization, and PReLU activation function, Cat(·) represents the concatenation operation in the channel dimension, Conv1(·) represents a 1×1 convolution, and Conv sc BConv3 represents a self-calibrated convolution, and BConv3 represents a 3×3 convolution kernel, batch normalization, and ReLU activation function.

[0029] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the efficient self-attention calibration decoding module uses convolutional layers to embed high-level features into overlapping patches, and models local continuity information by combining the information of each block and the blocks above, below, left, and right.

[0030] The obtained serialized tokens are used to compute the query tensor Q, key tensor K, and value tensor V through three parallel fully connected layers; the similarity between query Q and key tensor K is then calculated to obtain the similarity matrix QK. T Then, the similarity matrix QK T Divide by Scaling is performed, and the final weights are obtained. The sum tensor V is multiplied to obtain the output O. SRA ;

[0031] Output O SRA The result is added to V and then fed into the convolutional feedforward (CFF) module for further decoding to obtain the output O. CFF ;

[0032] Multiplying the query tensor Q and the key tensor K yields the fused feature Q. r K r Q is calibrated through two branches respectively. r K r Then, sequentially connect with output O CFF Multiplication is used to enhance global features, resulting in the first global feature O. CFF1 Second global feature O CFF2 The first branch is the channel attention module (CAM), and the second branch is the convolutional attention module (CIM).

[0033] The enhanced second global feature O CFF2 Upsampling is performed to obtain fine-grained detail features from the output. Multiply to supplement rich local details.

[0034] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the boundary enhancement and refinement module includes:

[0035] The gradient map of the overall direction is obtained by calculating the gradients of low-level feature X1 and high-level feature X4 in the horizontal and vertical directions using the Sobel operator.

[0036] The low-level feature X1 is convolved with the horizontal convolution matrix C. x1 and the vertical convolution matrix C y1 Convolution is performed to obtain the horizontal gradient map G. x1 and vertical gradient map G y1 Combined with the horizontal gradient map G x1 and vertical gradient map G y1 Obtain the gradient map in the global direction

[0037] Using the Sigmoid activation function on the gradient map Normalize the data and multiply it with the original feature X1 to obtain the boundary enhancement feature map. The same applies to the high-level feature X4;

[0038] Enhanced feature map of the boundary and boundary enhancement feature map Perform convolutional fusion operation to predict bladder tumor boundaries

[0039] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, the boundary enhancement feature map is... and boundary enhancement feature map The convolutional fusion operation utilizes two 1×1 convolutional units to reduce the boundary and enhance the feature map. and boundary enhancement feature map The number of channels was increased to 32; then bilinear upsampling was used to adjust... The size, so that it is consistent with The two layers are then concatenated and fused along the channel dimension. Boundary information is extracted and irrelevant noise is eliminated through two consecutive 3×3 convolutional units. Finally, a convolutional layer with a kernel size of 1 reduces the number of channels to 1, generating a bladder tumor boundary prediction map.

[0040] Preferably, in the above-mentioned bladder tumor image segmentation method based on FEBE-Net, the bladder tumor prediction map D output by the decoder at the five stages is... i Deep supervision is performed on i = 1, 2, 3, 4, b, and the generated boundary clues are analyzed. Conduct boundary surveillance;

[0041] The output of the boundary enhancement and refinement module Duplicate 4 times to obtain: Boundary Clues

[0042] Boundary clues Boundary Clues Boundary Clues Integrate the boundary cues into input features X4, D'4, and D'3, and use downsampling operations to extract the boundary cues respectively. Boundary Clues Boundary Clues The size is adjusted to be the same as that of input features X4, D'4, and D'3. Then, the downsampled boundary cue map is multiplied with input features X4, D'4, and D'3, and the result of the multiplication is added to input features X4, D'4, and D'3 to enhance the boundary regions in these high-level features.

[0043] Preferably, in the above-mentioned FEBE-Net-based bladder tumor image segmentation method, a hybrid supervision strategy is designed, and an overall loss function L is defined. overall To optimize the network, including:

[0044] Define the overall loss function L overall :

[0045]

[0046] L segi =L wiou (D i ,GT)+L wbce (D i ,GT)(i=1,2,3,4,b);

[0047]

[0048] In the above formula, The loss function for bladder tumor segmentation in five stages consists of weighted cross-linking loss and weighted binary cross-entropy loss. wiou and wbce constrain the bladder tumor prediction mask from global and local perspectives, respectively; L bnd The loss function for bladder tumor boundary segmentation is represented by the Dice loss function, where N represents the total number of pixels in the cystoscopy image, and g i and p i ω represents the ground truth value and predicted label of the i-th pixel, respectively; ω is the weighting factor that balances the segmentation loss and the boundary segmentation loss.

[0049] On the other hand, this invention discloses a bladder tumor image segmentation system based on FEBE-Net, which applies the above-mentioned method and includes:

[0050] The acquisition module acquires the RGB image to be segmented;

[0051] Build modules to construct the FEBE-Net prediction model;

[0052] The output module inputs the RGB image to be segmented into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

[0053] As can be seen from the above technical solution, compared with the prior art, this invention discloses a bladder tumor image segmentation method and system based on FEBE-Net, which can extract multi-scale global features and fine-grained detail features, and generate a bladder tumor prediction mask with clear boundaries. First, since the size and shape of bladder tumors vary greatly, a PVTv2 encoder is used to learn long-range dependencies. On the one hand, the proposed Feature Exploration Attention (FEA) module is used to explore local fine-grained detail information in the shallow features extracted by the PVTv2 encoder and eliminate noise. Then, it is supplemented into the corresponding high-level features at the decoder through lateral connection to fill semantic gaps. On the other hand, an efficient self-attention calibration decoding module (ESCD) is proposed to connect low-level features with high-level features, use an efficient self-attention mechanism to recover global features, and then fuse the calibrated and enhanced global features with fine-grained detail features to restore the local feature details of the tumor. Simultaneously, a Boundary Enhancement and Refinement (BER) module is proposed to capture boundary cues. On one hand, these cues are adaptively weighted and sent to the input features of the first three layers of the decoder to highlight and enhance boundary regions. On the other hand, they are added to the predicted output of the last layer of the decoder to achieve complementary fusion and generate a prediction map with clearer boundaries. Furthermore, a hybrid supervision strategy is designed to achieve dual supervision of the prediction mask and the boundary mask. Finally, to verify the segmentation performance of the proposed model, extensive experiments were conducted on the bladder tumor dataset BtAMU. The proposed method outperforms other state-of-the-art (SOTA) methods in all segmentation performance metrics. Specifically, on BtAMU, the MDice and MIou metrics of the proposed method reach 0.9126 and 0.8578, respectively. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0055] Figure 1 This is a schematic diagram of the FEBE-Net prediction model provided by the present invention;

[0056] Figure 2A schematic diagram of the feature exploration and interest module provided by this invention;

[0057] Figure 3 A schematic diagram of the high-efficiency self-attention calibration decoding module provided by the present invention;

[0058] Figure 4 A schematic diagram of the boundary enhancement and refinement module provided by the present invention;

[0059] Figure 5 This is a visualization of the qualitative results of the segmentation performance of FEBE-Net provided by this invention on the BTAMU bladder tumor dataset. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention. The embodiments of the present invention disclose a Transformer-based network architecture: FEBE-Net, for improving the segmentation accuracy of bladder tumors in cystoscopy. First, the overall architecture of the proposed FEBE-Net is briefly described. Then, the proposed Feature Exploration Attention Module (FEA), Efficient Self-Attention Calibration Decoding Module (ESCD), and Boundary Enhancement and Refinement Module (BER) are described in detail. Finally, the overall loss function used to train the network is introduced.

[0061] Specifically, the FEBE-Net prediction model is as follows: Figure 1 As shown, it includes a PVT v2 encoder and three proposed modules: Feature Exploration Attention (FEA) module, Efficient Self-Attention Calibration Decoding (ESCD) module, and Boundary Enhancement and Refinement (BER) module.

[0062] Specifically, given an input RGB image X∈R to be segmented... H×W×3 Because visual transformers are more capable than CNNs of capturing long-range dependencies and feature representations, the PVT v2 encoder was chosen as the backbone network to capture multi-scale global feature information of bladder tumors, adapting to changes in tumor shape and size. The Transformer backbone network PVT v2 extracts pyramid features X at four different levels. i (i = 1, 2, 3, 4), where X1, X2, X3 are low-level features, which contain more fine-grained details and noise information that is irrelevant to bladder tumor segmentation, and X4 is a high-level semantic feature, which contains more feature information that can locate bladder tumors.

[0063] Then, the low-level feature Xi (i=1,2,3) are sent to the FEA module to mine potential local detailed features of bladder tumors and filter out irrelevant noise to generate feature maps. Simultaneously, low-level feature X1 and high-level feature X4 are input into the BER module to generate high-quality boundary cues. And Copy four times to get Next, high-level features X4 and boundary cues It is sent to the decoder, and first used Enhance the boundary regions in X4, then recover global information through a 1×1 convolutional unit to obtain feature map D'4. D'4 is then passed through another 1×1 convolution to reduce the number of channels to 1 to generate the initial global prediction map D4; then, Send to the proposed decoder module ESCD, utilizing boundary clues For high-level features D' i+1 (i=2,3) are adaptively weighted to gradually guide the network to focus on noteworthy bladder tumor boundary regions. Through the lateral connection between the FEA module and the ESCD module, feature maps containing rich local details are transferred. The high-level feature map D' recovered from the decoder in the previous stage i (i=2,3,4) are fused together, and a more accurate bladder tumor prediction map D is gradually generated through deep decoding. i (i = 1, 2, 3); To further refine the boundary regions in the prediction map, we utilize... Fine-tuning of the prediction graph D1 to achieve Complementing the feature information in D1, a more clearly defined and accurate bladder tumor prediction map D is obtained. b .

[0064] During the experiment, the bladder tumor prediction map D output from the five stages at the decoder end was analyzed. i Deep supervision is performed on (i = 1, 2, 3, 4, b), and the generated boundary clues are evaluated. Conduct boundary monitoring.

[0065] The FEBE-Net prediction model structure is defined as follows:

[0066] X i =PVTv2(X), (i = 1, 2, 3, 4);

[0067]

[0068]

[0069] D i =Conv1(D' i), (i = 1, 2, 3, 4);

[0070]

[0071] To further optimize the above technical solution, the Feature Exploration and Focus Module (FEA) includes:

[0072] Since the low-level features X1, X2, and X3 extracted by the PVT v2 encoder contain rich detailed feature information, these features can be explored and focused upon to supplement the high-level features with the captured fine-grained details, thus addressing the issues of lost local detailed feature information and small bladder tumors in high-level features. However, these low-level features from the encoder end contain more background noise and irrelevant information, creating a semantic gap between them and the high-level features. Therefore, a Feature Exploration and Focus Module (FEA) is proposed, aiming to extract local fine-grained details from shallow features while filtering noise, and laterally connect it between the encoder and decoder to bridge the semantic gap between low-level and high-level features.

[0073] The proposed FEA module structure is as follows: Figure 2 As shown, the FEA module has four branches, each performing different convolution operations on low-level features X1, X2, and X3 to capture local information. The first branch uses a 3×3 convolutional unit with a scaling factor of 1 for feature mining. The second branch adds another 3×3 convolutional unit with a scaling factor of 3 for further feature exploration, and so on. The third branch uses three 3×3 convolutional units with scaling factors of 1, 3, and 5 to supplement and explore features. The outputs of the first three branches are then concatenated along the channel dimension to expand the receptive field, enabling more effective identification of large bladder tumors. Next, a 1×1 convolution is used to reduce the number of channels and decrease computational resources. Unlike the first three branches, the fourth branch uses self-calibrating convolutions to capture richer detailed features, enhancing the identification of small bladder tumors. This is then followed by concatenation, further enhancing features and capturing fine-grained details. Subsequently, a Linear Efficient Channel Focus Module (LECA) is used to filter noise and purify feature information, forcing the network to focus on features closely related to bladder tumor segmentation. Finally, a 3×3 convolutional unit is used to further denoise the features and restore them to the original number of channels. In this way, the FEA module can uncover potential local detail features of bladder tumors and enhance feature representation. This process can be described as follows:

[0074] F 11 =DConv1(X i ), (i = 1, 2, 3);

[0075] F 12=DConv3(DConv1(X i ),(i=1,2,3);

[0076] F 13 =DCon5(DConv3(DConv1(X) i ))),(i=1,2,3);

[0077] F1 = Conv1(Cat(F 11 ,F 12 ,F 13 ));

[0078] F2 = Conv sc (X i ), (i = 1, 2, 3);

[0079] X Fi =BConv3(LECA(Cat(F1,F2)));

[0080] In the above formula, DConvi(i=1,3,5) represents a 3×3 convolution kernel with expansion ratios of 1, 3, and 5, respectively, batch normalization, and PReLU activation function, Cat(·) represents the concatenation operation in the channel dimension, Conv1(·) represents a 1×1 convolution, and Conv sc BConv3 represents a self-calibrated convolution, with BConv3 indicating a 3×3 convolution kernel, batch normalization, and ReLU activation function.

[0081] To further optimize the above technical solution, the high-efficiency self-attention calibration decoding module (ESCD) includes:

[0082] Since high-level features typically contain more semantic information that can identify bladder tumor regions, they may lose some detailed information about bladder tumors or small bladder tumors. Therefore, an efficient self-attention calibration decoding module (ESCD) is proposed, which integrates the fine-grained details captured by the FEA module in low-level features with the global semantic information in high-level features, so as to effectively restore the local feature details of the tumor while achieving accurate bladder tumor localization.

[0083] The proposed ESCD module has the following structure: Figure 3As shown, firstly, high-level features are embedded into overlapping patches using convolutional layers with a stride of 4 and a kernel size of 4. Local continuity information is modeled by combining information from each block and its adjacent blocks (top, bottom, left, and right). Then, the obtained serialized token is processed through three parallel fully connected layers to compute the query tensor Q, key tensor K, and value tensor V, respectively. Next, the obtained serialized token is processed through three parallel fully connected layers to compute the query tensor Q, key tensor K, and value tensor V, respectively; the similarity between query Q and key tensor K is calculated to obtain the similarity matrix QK. T Then, the similarity matrix QK T Divide by Scaling is performed, and the final weights are obtained. The sum tensor V is multiplied to obtain the output O. SRA This process can be described by a formula:

[0084]

[0085] In the formula, Indicates the dimension of the feature embedding. O SRA The result is added to V and then fed into the convolutional feedforward (CFF) module for further decoding to obtain the output O. CFF This process can be described by formula (16-17):

[0086] O CFF =O SRA +V+CFF(O SRA +V);

[0087] CFF(x)=fc2(R(DWConv(fc1(x))));

[0088] In the formula, x is the input feature, fc1(·) and fc2(·) are linear fully connected mapping layers with an inflation rate of 4, DWConv(·) is a deep convolutional layer, which is used to model positional information and reduce computation, and R(·) represents the ReLU activation function.

[0089] Since the query tensor Q and the key tensor K carry inherent contextual statistical information, multiplying the query tensor Q and the key tensor K yields the fused feature Q. r K r Q is calibrated through two branches respectively. r K r This filters out more valuable contextual information, which is then sequentially compared with O. CFFMultiplication is used to enhance global features and eliminate noise. The first branch is the Channel Attention Module (CAM), and the second branch is the Convolutional Attention Module (CAM), which consists of a 3×3 convolutional unit and a sigmoid function. Due to the similarity between K and V, these two branches can weaken irrelevant contextual information in the channel dimension and spatial dimension, respectively. This process can be described by the following formula:

[0090] Q r K r = r(Q) × r(K)

[0091] O CFF1 =CAM(Q r K r )×O CFF ;

[0092] O CFF2 =S(BConv3(Q) r K r ))×O CFF1 ;

[0093] In the above formula, r(·) represents the feature reshape operation, BConv3 represents the 3×3 convolution kernel, batch normalization and ReLU activation function, and S(·) represents the Sigmoid activation function.

[0094] Then, the enhanced global feature O CFF2 Upsampling is performed, and the fine-grained detail features output by the FEA module are compared. Multiplication, to supplement rich local detail features, helps the decoder module restore fine-grained details when predicting bladder tumor regions. This process can be described by a formula:

[0095]

[0096] To further optimize the above technical solution, the boundary enhancement and refinement module (BER) includes:

[0097] Since boundary feature information can provide useful fine-grained features, it can help enhance the learning ability of semantic segmentation of medical images. Therefore, a boundary enhancement and refinement module (BER) is proposed. It directly utilizes the first and fourth layer features extracted by the encoder to generate high-quality boundary cues, and then integrates them into the bladder tumor prediction task. The aim is to enhance the boundary regions in the high-level features at the decoder end and fine-tune the final prediction map, so that the network generates prediction masks with more accurate localization and clearer boundaries.

[0098] The proposed boundary enhancement and refinement module has the following structure: Figure 4As shown, boundary feature information is extracted from low-level feature X1 and high-level feature X4, and boundary-irrelevant information is filtered out.

[0099] First, the gradient map of the overall direction is obtained by calculating the gradients of low-level feature X1 and high-level feature X4 in the horizontal and vertical directions using the Sobel operator. Specifically, taking low-level feature X1 as an example, the convolutions C in the horizontal and vertical directions... x1 and C y1 They are defined as follows:

[0100]

[0101] Then, X1 and C respectively x1 and C y1 Convolution is performed to obtain gradient maps G in the horizontal and vertical directions. x1 and G y1 Combined with G x1 and G y1 Obtain the gradient map in the global direction

[0102]

[0103] Next, the gradient map is processed using the Sigmoid activation function. Normalize the data and multiply it with the original feature X1 to obtain the boundary enhancement feature map. Similarly, for the high-level feature X4, we can obtain:

[0104]

[0105]

[0106] In the formula, S(·) represents the Sigmoid activation function.

[0107] Then, the boundary enhancement feature map and A series of convolutional fusion operations are performed; specifically, two 1×1 convolutional units are used to reduce... and The number of channels was increased to 32, which reduced the consumption of computing resources while further extracting features; then, a bilinear upsampling operation was used to adjust... The size, so that it is consistent with The two layers are then concatenated and fused along the channel dimension. Boundary information is further extracted and irrelevant noise is eliminated using two consecutive 3×3 convolutional units. Finally, a convolutional layer with a kernel size of 1 reduces the number of channels to 1, generating a bladder tumor boundary prediction map. This process can be described by the following formula:

[0108]

[0109]

[0110] In the above formula, BConv1 represents a 1×1 convolution kernel, batch normalization, and ReLU activation function. Cat(·) represents the connection operation in the channel dimension, BConv3 represents a 3×3 convolution kernel, batch normalization, and ReLU activation function, and Conv1(·) represents a 1×1 convolution.

[0111] Simultaneously, the same Sobel algorithm is applied to each bladder tumor mask in the ground truth image (GT) to generate the corresponding tumor boundary mask, resulting in a ground truth boundary image (GTB). The GTB is then used to analyze... Boundary supervision can eliminate edge feature information inside bladder tumors and obtain high-quality bladder tumor boundary clues.

[0112] The output of the BER module Copy 4 times to get: Because the input features X4, D4, and D3 of the first three layers at the decoder contain more high-level semantic concepts, however, there is no clear boundary between foreground and background, therefore, Integrating into high-level features X4, D4, and D3, specifically, using downsampling operations to extract boundary cues. The size of the boundary cue map is adjusted to be the same as X4, D4, D3. Then, the downsampled boundary cue map is multiplied with X4, D4, D3, and the result is added to X4, D4, D3 to enhance the boundary regions in these high-level features. This aims to progressively guide the network to focus on noteworthy boundary regions, thereby generating a more accurate bladder tumor prediction map. Furthermore, it utilizes... The final predicted feature map is fine-tuned to achieve information complementarity and further refine the boundary region of the bladder tumor prediction map.

[0113] like Figure 1 As shown, the proposed FEBE-Net requires the ground truth map (GT) and ground truth boundary map (GTB) to respectively modify the bladder tumor prediction mask D. i (i = 1, 2, 3, 4, b) and tumor boundary prediction mask To ensure effective supervision, a hybrid supervision strategy was designed, and an overall loss function L was defined. overall To optimize the network:

[0114]

[0115] In the above formula, The loss function for bladder tumor segmentation in five stages is represented by a weighted cross-linking loss (wiou) and a weighted binary cross-entropy loss (wbce). wiou and wbce constrain the bladder tumor prediction mask from global and local perspectives, respectively. bnd The loss function used for bladder tumor boundary segmentation is the Dice loss function, which can mitigate the negative impact of area imbalance between foreground and background pixels in boundary segmentation. N represents the total number of pixels in the cystoscopy image, and g... i and p i Let represent the ground truth value and predicted label of the i-th pixel, respectively. ω is a weighting factor that balances the segmentation loss and the boundary segmentation loss; in this paper, it is set to 5.

[0116] First, the dataset, evaluation metrics, and implementation details are introduced. The results of the proposed FEBE-Net are compared with state-of-the-art (SOTA) methods to demonstrate its superiority. Ablation experiments are conducted to demonstrate the effectiveness of the three proposed novel modules.

[0117] Bladder tumor dataset BtAMU:

[0118] BtAMU consists of 1948 bladder tumor images extracted from 110 4K cystoscopy videos provided by the Department of Urology, First Affiliated Hospital of Anhui Medical University, along with their corresponding ground truth labels. The ground truth labels were manually annotated by professional urologists. The image resolution is 1920×1080. In the bladder tumor segmentation experiment, 1562 images from the BtAMU dataset were randomly selected in an 8:2 ratio as the training dataset, and the remaining 386 images were used as the test dataset.

[0119] In bladder tumor segmentation experiments, the following parameters were selected: mean Dice coefficient (mDice), mean crossover ratio (mIoU), mean absolute error (MAE), accuracy, and weighted average. Hausdorff distance (HD) was used as an evaluation metric for quantitative analysis. In the polyp segmentation task, mDice, mIoU, and HD were used as evaluation metrics. mDice and mIoU are region-level similarity metrics, representing the degree of agreement between the model's predictions and the actual results; MAE is a pixel-level evaluation metric, representing the mean absolute error between the algorithm's predictions and the actual values; Accuracy represents the proportion of pixels correctly predicted by the model out of the actual pixels. It is a metric that comprehensively considers precision and recall, using a coefficient β to balance precision and recall; HD is used to measure the segmentation accuracy at the boundary. Among them, mDice, mIoU, Accuracy, and Higher values ​​for the metrics indicate better algorithm prediction performance, while lower values ​​for MAE and HD indicate better model segmentation performance. Their definitions are as follows:

[0120]

[0121] In the above formula, TP represents true positive, TN represents true negative, FP represents false positive, and FN represents false positive; N represents the number of test images. In the formula, P represents the predicted image, Y represents the ground truth, and W and H are the width and height of the image, respectively; β is a parameter representing precision. ω and Recall ω Let represent weighted precision and weighted recall, respectively. In this paper, β is set to 1. In the formula, A and B represent the ground truth image and the predicted image, respectively, and ||ab|| represents the distance function, such as the Euclidean distance function.

[0122] The proposed FEBE-Net is implemented using the PyTorch framework, with 24GB of RAM and an NVIDIA GeForce RTX 3090 GPU for accelerated computation. The optimizer is set to AdamW, and the initial learning rate is set to 5×10⁻⁶. -5 The weight decay rate was set to 0.1, the batch size to 8, and the training lasted for 100 epochs. To accelerate network convergence, the learning rate was decreased by 0.5 after the 15th and 30th epochs. Furthermore, the input image size was resized to 352×352, and a multi-scale training strategy with gradient clipping constrained to 0.5 ({0.75, 1.0, 1.25}) was used.

[0123] To verify the effectiveness of the proposed FEBE-Net for bladder tumor segmentation, the quantitative results of the proposed FEBE-Net were compared with those of UNet, PraNet, CaraNet, HarDNet-MSEG, DCRNet, MSRAformer, HSNet, and TGDAUNet, and the results were evaluated using mDice, mIoU, MAE, and accuracy. Six medical image segmentation evaluation metrics, including HD, were used to quantitatively evaluate the segmentation performance of all models on the BtAMU bladder tumor dataset. Table 1 lists the quantitative results of different algorithms on the BtAMU bladder tumor dataset. As can be seen from Table 1, compared with other models, the proposed FEBE-Net achieves state-of-the-art segmentation performance on the BtAMU dataset. In terms of mDice and mIoU metrics, FEBE-Net improves by 1.4% and 1.63% respectively compared to the second-best performing MSRAformer, indicating that FEBE-Net can better distinguish bladder tumors from normal tissue background. Similarly, FEBE-Net's accuracy and... It also improved performance by 0.16% and 0.52% compared to the second-best performing MSRAformer, respectively. Furthermore, FEBE-Net reduced MAE and HD metrics by 0.22% and 0.3454 compared to the second-best performing HSNet, respectively. Quantitative HD results indicate that FEBE-Net improved the accuracy of bladder tumor boundary segmentation and reduced boundary segmentation errors. These results validate the effectiveness and superiority of the proposed FEBE-Net for bladder tumor segmentation in cystoscopic images.

[0124] Table 1

[0125]

[0126] A comparison of quantitative results of the FEBE-Net-based prediction model on the BtAMU dataset with other state-of-the-art (SOTA) methods. ↑ indicates higher is better, ↓ indicates lower is better, and bold indicates the best performance.

[0127] To more clearly and intuitively demonstrate the effectiveness of the proposed FEBE-Net, Figure 5 The visualization results of FEBE-Net on the BtAMU bladder tumor dataset are presented. From top to bottom, the images show the original bladder tumor image, the ground truth (GT) image, the bladder tumor mask predicted by FEBE-Net, and a comparison of the FEBE-Net prediction image with the GT. In the fourth row, white represents regions predicted by FEBE-Net and presented in the GT (TP, correctly predicted, predicted result is positive, and the true value is positive). Blue represents regions predicted by BT-PVT but not present in the GT (FP, incorrectly predicted, predicted result is positive, and the true value is negative). Red regions represent false detections. Red regions represent regions present in the GT but not predicted by BT-PVT (FN, incorrectly predicted, predicted result is negative, and the true value is positive), which are missed detections. Figure 5 As can be seen, FEBE-Net can effectively address the challenges posed by the varying shapes and sizes of bladder tumors and their uneven brightness.

[0128] Specific Figure 5 The proposed FEBE-Net segmentation performance on the BTAMU bladder tumor dataset is visualized qualitatively. The first row shows bladder tumor images of different shapes and sizes under cystoscopy; the second row shows the ground truth (GT) images of the BTAMU bladder tumor dataset; the third row shows the bladder tumor mask predicted by FEBE-Net; and the fourth row shows the visualization results comparing the FEBE-Net prediction map with the GT.

[0129] In the proposed FEBE-Net, three new modules (FEA, ESCD, BER) are introduced to improve bladder tumor segmentation performance. To verify the effectiveness of each module, ablation experiments are conducted on the bladder tumor dataset BT-AMU to explore the impact of each module on bladder tumor segmentation performance.

[0130] Specifically, the baseline was PVT v2; to verify the effectiveness of the FEA module, it was directly removed; for the ESCD module, it was replaced with a 1×1 convolutional unit and a 2x upsampling operation for comparison; for the BER module, it was directly removed to verify its effectiveness. The ablation experiments of the above FEA, ESCD, and BER modules were labeled "w / o FEA", "w / o ESCD", and "w / o BER", respectively. The experimental results are shown in Table 2. Table 2 shows that replacing or removing any of the proposed modules significantly reduces the segmentation performance of bladder tumors, demonstrating the effectiveness of each proposed module; the segmentation performance of "w / o FEA", "w / o ESCD", and "w / o BER" ​​is higher than the baseline, reflecting the effectiveness of cooperation between the modules.

[0131] Table 2

[0132]

[0133] Quantitative results of FEBE-Net ablation experiments on the BTAMU bladder tumor dataset. ↑ indicates higher is better, ↓ indicates lower is better, and bold black indicates the best performance.

[0134] In summary, the Feature Exploration and Focus Module (FEA) mines potential details in low-level features through different convolution operations, solving the problems of lost local details and small bladder tumors in high-level features. Furthermore, the LECA mechanism and lateral connectivity help bridge the semantic gap between features, improving the segmentation accuracy of bladder tumors.

[0135] The efficient self-attention calibration decoding module (ESCD) effectively recovers global features and fuses local detail features with the calibrated and enhanced global features, effectively restoring detail features, enhancing the model's feature representation ability, and solving the problem that Transformer cannot accurately represent local details.

[0136] The Boundary Enhancement and Refinement (BER) module effectively extracts bladder tumor boundaries through boundary detection operators. This not only provides valuable boundary clues for the decoder to predict bladder tumor regions but also refines and adjusts the final bladder tumor prediction map. Unlike existing methods, BER directly uses boundary supervision to explore boundary clues and integrates the bladder tumor boundary extraction task with the tumor prediction task, thereby improving the learning ability of bladder tumor semantic segmentation.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0138] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A bladder tumor image segmentation method based on FEBE-Net, characterized in that, include: Obtain the RGB image to be segmented; The FEBE-Net prediction model was constructed, and the PVTv2 encoder was selected as the backbone network to capture multi-scale global feature information of bladder tumors. The PVTv2 encoder extracted pyramid features X at four different levels. i , i = 1, 2, 3, 4; The low-level feature X i The numbers i=1,2,3 are sent to the feature exploration and focus module to mine potential local detailed features of bladder tumors and filter out irrelevant noise, generating feature maps. i = 1, 2, 3; simultaneously, low-level feature X1 and high-level feature X4 are input into the boundary enhancement and refinement module to generate high-quality boundary cues. And Copy four times to get Advanced features X4 and boundary cues The data is sent to the decoder and first uses boundary clues. Enhance the boundary regions in high-level feature X4, then recover global information through a 1×1 convolutional unit to obtain feature map D'4. Feature map D'4 is then passed through a 1×1 convolution to reduce the number of channels to 1 to generate the initial global prediction map D4. Boundary Clues Advanced Feature D' i+1 The values ​​i = 2, 3, and 4 are sent to the proposed efficient self-attention calibration decoding module, utilizing boundary cues. i = 2, 3 pairs of high-level features D' i+1 Adaptive weighting is applied to i=2 and i=3 to gradually guide the network to focus on noteworthy bladder tumor boundary regions. Through the lateral connection between the feature exploration and attention module and the efficient self-attention calibration decoding module, feature maps containing rich local details are generated. The high-level feature map D' recovered from the decoder in the previous stage i The bladder tumor prediction map D is generated step by step through deep decoding, where i = 2, 3, and 4 are fused. i , i = 1, 2, 3; bladder tumor prediction map D output from the 5 stages of the decoder. i Deep supervision is performed on i = 1, 2, 3, 4, b, and the generated boundary clues are analyzed. Boundary supervision is performed, and a hybrid supervision strategy is designed to define the overall loss function L. overall To optimize the network; The RGB image to be segmented is input into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

2. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that, The mathematical expression for the FEBE-Net prediction model is: X i =PVTv2(X),(i=1,2,3,4); D i =Conv1(D′ i ),(i=1,2,3,4); 3. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that, The feature exploration and attention module comprises four branches. Each branch performs different convolution operations on low-level features X1, X2, and X3 to capture local information. The first branch uses a 3×3 convolutional unit with a scaling factor of 1 for feature mining. Based on this, the second branch adds another 3×3 convolutional unit with a scaling factor of 3 for further feature exploration, and so on. The third branch supplements and explores features using three 3×3 convolutional units with scaling factors of 1, 3, and 5, respectively. The outputs of the first three branches are then concatenated along the channel dimension, using a 1×1 convolution to reduce the number of channels. The fourth branch uses a self-calibrating convolution to capture richer detailed features, followed by concatenation. A linear and efficient channel attention module is used to filter noise. Finally, a 3×3 convolutional unit is used to further denoise the features and restore the original number of channels.

4. The bladder tumor image segmentation method based on FEBE-Net according to claim 3, characterized in that, The mathematical expression for the feature exploration focus module is: F 11 =DConv1(X i ),(i=1,2,3); F 12 =DConv3(DConv1(X i )),(i=1,2,3); F 13 =DConv5(DConv3(DConv1(X i ))),(i=1,2,3); F1=Conv1(Cat(F 11 ,F 12 ,F 13 )); F2=Conv sc (X i ),(i=1,2,3); X Fi =BConv3(LECA(Cat(F1,F2))); In the above formula, DConvi(i=1,3,5) represents a 3×3 convolution kernel with expansion ratios of 1, 3, and 5, respectively, batch normalization, and PReLU activation function, Cat(·) represents the concatenation operation in the channel dimension, Conv1(·) represents a 1×1 convolution, and Conv sc BConv3 represents a self-calibrated convolution, and BConv3 represents a 3×3 convolution kernel, batch normalization, and ReLU activation function.

5. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that, The efficient self-attention calibration decoding module uses convolutional layers to embed high-level features into overlapping patches, and models local continuity information by combining information from each block and its adjacent blocks above, below, left, and right. The obtained serialized tokens are used to compute the query tensor Q, key tensor K, and value tensor V through three parallel fully connected layers; the similarity between query Q and key tensor K is then calculated to obtain the similarity matrix QK. T Then, the similarity matrix QK T Divide by Scaling is performed, and the final weights are obtained. The sum tensor V is multiplied to obtain the output O. SRA ; Output O SRA The result is added to V and then fed into the convolutional feedforward (CFF) module for further decoding to obtain the output O. CFF ; Multiplying the query tensor Q and the key tensor K yields the fused feature Q. r K r Q is calibrated through two branches respectively. r K r Then, sequentially connect with output O CFF Multiplication is used to enhance global features, resulting in the first global feature O. CFF1 Second global feature O CFF2 The first branch is the channel attention module (CAM), and the second branch is the convolutional attention module (CIM). The enhanced second global feature O CFF2 Upsampling is performed to obtain fine-grained detail features from the output. Multiply to supplement rich local details.

6. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that, The boundary enhancement and refinement module includes: The gradient map of the overall direction is obtained by calculating the gradients of low-level feature X1 and high-level feature X4 in the horizontal and vertical directions using the Sobel operator. The low-level feature X1 is convolved with the horizontal convolution matrix C. x1 and the vertical convolution matrix C y1 Convolution is performed to obtain the horizontal gradient map G. x1 and vertical gradient map G y1 Combined with the horizontal gradient map G x1 and vertical gradient map G y1 Obtain the gradient map in the global direction Using the Sigmoid activation function on the gradient map Normalize the data and multiply it with the original feature X1 to obtain the boundary enhancement feature map. The same applies to the high-level feature X4; Enhanced feature map of the boundary and boundary enhancement feature map Perform convolutional fusion operation to predict bladder tumor boundaries 7. The bladder tumor image segmentation method based on FEBE-Net according to claim 6, characterized in that, Enhanced feature map of the boundary and boundary enhancement feature map The convolutional fusion operation utilizes two 1×1 convolutional units to reduce the boundary and enhance the feature map. and boundary enhancement feature map The number of channels was increased to 32; then bilinear upsampling was used to adjust... The size, so that it is consistent with The two layers are then concatenated and fused along the channel dimension. Boundary information is extracted and irrelevant noise is eliminated through two consecutive 3×3 convolutional units. Finally, a convolutional layer with a kernel size of 1 reduces the number of channels to 1, generating a bladder tumor boundary prediction map.

8. The bladder tumor image segmentation method based on FEBE-Net according to claim 1, characterized in that, Bladder tumor prediction map D output from the five stages of the decoder. i Deep supervision is performed on i = 1, 2, 3, 4, b, and the generated boundary clues are analyzed. Conduct boundary surveillance; The output of the boundary enhancement and refinement module Duplicate 4 times to obtain: Boundary Clues Boundary clues Boundary Clues Boundary Clues Integrating into input features X4, D'4, and D'3, and using downsampling operations to extract boundary cues respectively. Boundary Clues Boundary Clues The size is adjusted to be the same as that of input features X4, D'4, and D'3. Then, the downsampled boundary cue map is multiplied with input features X4, D'4, and D'3, and the multiplication result is added to input features X4, D'4, and D'3 to enhance the boundary regions in input features X4, D'4, and D'3.

9. The bladder tumor image segmentation method based on FEBE-Net according to claim 6, characterized in that, Design a hybrid supervision strategy and define the overall loss function L. overall To optimize the network, including: Define the overall loss function L overall : In the above formula, (i = 1, 2, 3, 4, b) represents the loss function for bladder tumor segmentation in 5 stages, consisting of weighted cross-linking loss and weighted binary cross-entropy loss. wiou and wbce constrain the bladder tumor prediction mask from global and local perspectives, respectively; L bnd The loss function for bladder tumor boundary segmentation is represented by the Dice loss function, where N represents the total number of pixels in the cystoscopy image, and g i and p i ω represents the ground truth value and predicted label of the i-th pixel, respectively; ω is the weighting factor that balances the segmentation loss and the boundary segmentation loss.

10. A bladder tumor image segmentation system based on FEBE-Net, employing the bladder tumor image segmentation method based on FEBE-Net according to any one of claims 1-9, characterized in that, include: The acquisition module acquires the RGB image to be segmented; Build modules to construct the FEBE-Net prediction model; The output module inputs the RGB image to be segmented into the FEBE-Net prediction model to obtain a bladder tumor prediction map.

Images