A boundary auxiliary kidney tumor segmentation method based on a transformer

Abstract

The application discloses a boundary auxiliary kidney tumor segmentation method based on a Transformer, and is applied to the technical field of kidney tumor segmentation and comprises the following steps: a PVTv2 backbone extraction network is used to extract backbone features in an endoscope image, and an adaptive context extraction module is used to adaptively extract the backbone features in different stages; a mixed feature capture module is used to mix the features obtained after adaptive extraction in different stages with output features of a previous-stage mixed feature capture module, and the features are decoded through an SA-FF module; a boundary auxiliary module is used to extract boundary contour features in the backbone features, and the boundary contour features are fused with features finally output by the mixed feature capture module to obtain a boundary auxiliary kidney tumor segmentation prediction graph. The application can effectively extract local detail information, and solve the feature misplacement problem during fusion of high-level semantics and low-level semantics and the boundary blur problem of the kidney tumor.

Description

Technical Field

[0001] This invention relates to the field of kidney tumor segmentation technology, and in particular to a boundary-assisted kidney tumor segmentation method based on Transformer. Background Technology

[0002] Renal cell carcinoma is a common malignant tumor that seriously endangers human health. Kidney tumors are precursors to renal cell carcinoma and, if left untreated, can easily transform into malignant tumors. Laparoscopic partial resection of kidney tumors is currently the main method for treating kidney tumors. Segmenting endoscopic images helps doctors better identify and remove lesions; therefore, automatic and accurate segmentation of kidney tumors in endoscopic images is of great clinical significance.

[0003] Traditional kidney tumor segmentation methods typically rely on manually extracted features to identify kidney tumors, such as texture analysis, color distribution, geometric features, and intensity distribution. In recent years, with the continuous development of deep learning, deep learning-based medical image segmentation methods have proven superior to traditional manual feature extraction methods. With the introduction of U-Net, encoder-decoder networks have become the mainstream network architecture in medical image segmentation. Classic deep learning algorithms include network models such as U-Net, U-Net++, and ResNet++. However, the inherent encoder-decoder structure of convolutional operators causes the model to lose some global appearance details during training, making it difficult to establish global dependencies between features. Therefore, a method capable of extracting global features is needed to assist in image segmentation tasks.

[0004] Compared to CNNs, Transformers focus more on global information in images and have better generalization capabilities. They divide images into a number of small patches, learn the differences between these patches through self-attention, and model the correlations between different patches, more effectively extracting long-range dependencies. This allows the network to understand semantic differences from a global perspective, helping the model to more reliably perceive global contextual features and improve its ability to locate lesions and extract boundaries. However, Transformers cannot accurately extract local details. During feature fusion, simply combining low-resolution high-level features with high-resolution low-level features can lead to severe feature misalignment. Furthermore, Transformer segmentation results also have some issues. First, due to the variable size, shape, and location of kidney tumors, current kidney tumor segmentation methods still have limitations in extracting multi-scale features. Also, due to interference from blood vessels, fat, etc., the boundaries of kidney tumors are not clear, and their texture and color are highly similar to surrounding tissues, making them difficult to identify.

[0005] Therefore, how to provide a Transformer-based boundary-assisted renal tumor segmentation method that can effectively extract local detail information and solve the feature misalignment problem during the fusion of high-level and low-level semantics as well as the boundary ambiguity problem of renal tumors is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention proposes a boundary-assisted renal tumor segmentation method based on Transformer.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A Transformer-based boundary-assisted renal tumor segmentation method includes:

[0009] Step (1): The PVTv2 backbone extraction network is used to extract the backbone features in the endoscopic image, and the adaptive context extraction module is used to adaptively extract the backbone features at different stages.

[0010] Step (2): The hybrid feature capture module is used to mix the features obtained after adaptive extraction at different stages with the output features of the previous hybrid feature capture module, and then decodes them through the SA-FF module;

[0011] Step (3): The boundary contour features in the backbone features are extracted using the boundary assistance module and fused with the features finally output by the hybrid feature capture module to obtain the boundary-assisted renal tumor segmentation prediction map.

[0012] Optionally, in step (1), the backbone feature is a multi-scale pyramid feature.

[0013] Optionally, in step (1), the adaptive context extraction module is used to adaptively extract the backbone features, specifically as follows:

[0014] The backbone features are subjected to dilated convolution with different dilation rates to obtain feature maps containing multi-scale contextual information, and then the feature maps are multiplied together.

[0015] The SA+ module is used to perform convolution and transpose convolution on the multiplied feature maps respectively to obtain two feature maps, which are then reshaped.

[0016] Matrix multiplication is performed between the two feature maps, and a softmax operation is performed to compute the spatial attention map, which is then multiplied with the feature map obtained by transposing the convolution.

[0017] Optionally, in step (2), the mixed features are decoded using the SA-FF module, specifically:

[0018] The mixed features are divided into Q, K and V after passing through three fully connected layers and then fed into the SA-FF module for decoding. The AC module efficiently extracts Q and K and assists the output of the SA-FF module.

[0019] Optionally, in step (2), the SA-FF module includes the attention module and forward propagation module inherent in the Transformer, as follows:

[0020] I SA-FF =BN(O SRA +V)+MLP(BN(O SRA +V))

[0021] The MLP module is as follows:

[0022] O MLP =ReLU(f c2 (DW(ReLU(f c1 (x)))));

[0023] Where x is the input; f c1 (), f c2 () represents a convolution operation; DW represents a depthwise convolution; ReLU represents an activation function.

[0024] Optionally, the AC module performs efficient extraction of Q and K, as follows:

[0025]

[0026] in, This is a 3x3 convolution operation.

[0027] Optionally, in step (3), the boundary auxiliary module is used to extract the boundary contour features from the backbone features, specifically:

[0028] Location information is extracted using spatial attention mechanisms, as follows:

[0029]

[0030] in, This is a 3x3 convolution operation; This is a 1x1 convolution operation;

[0031] O S Input the RC module as follows:

[0032]

[0033]

[0034]

[0035] in, This is a 3x3 convolution operation; This is a 1x1 convolution operation.

[0036] Optionally, in step (3), the boundary contour features are fused with the features finally output by the hybrid feature capture module, as follows:

[0037]

[0038]

[0039]

[0040]

[0041] in, This is a 1x1 convolution operation; This is a 3x3 convolution operation; This is a 1x3 convolution operation; This is a 3x1 convolution operation; This is a 1x5 convolution operation; This is a 5x1 convolution operation.

[0042] Optionally, it also includes: using the outputs of the hybrid feature capture module and the boundary assistance module as the optimization objective, and defining a loss function by combining the weighted intersection-union loss function and the weighted binary cross-entropy loss function, as follows:

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Among them, y i and y' i These are the ground truth value of the i-th pixel in the n-th stage output and the final output of the frame, respectively. and These are the weighted binary cross-entropy loss and the weighted intersection-union loss, respectively.

[0049] As can be seen from the above technical solutions, compared with the prior art, this invention proposes a boundary-assisted renal tumor segmentation method based on Transformer. By using a PVTv2 backbone extraction network to extract more robust backbone features from endoscopic images, more foreground information is provided for the subsequent decoding stage, exhibiting stronger global information extraction capabilities and better robustness to input interference. The adaptive context extraction module employs multiple convolutions with different dilation rates to acquire a wider range of features from different receptive domains, effectively extracting local detail information and solving the problem of multi-scale feature adaptation in renal tumors. Furthermore, it can remove background noise interference and reduce misjudgments. The hybrid feature capture module (MFC) addresses the fusion difference between high-level and low-level semantics, narrowing the semantic gap and effectively recovering features. The boundary assistance module (BA) extracts boundary contour features and compares and fuses them with the output of the hybrid feature capture module (MFC) to enhance boundary contour details. In summary, this invention can effectively extract local detail information and solve the feature misalignment problem during the fusion of high-level and low-level semantics, as well as the boundary blurring problem of renal tumors. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0052] Figure 2 This is a schematic diagram of the network structure of the boundary-assisted renal tumor segmentation method based on Transformer of the present invention.

[0053] Figure 3 This is a schematic diagram of the adaptive context extraction module structure of the present invention.

[0054] Figure 4 This is a schematic diagram of the hybrid feature capture module structure of the present invention.

[0055] Figure 5 This is a schematic diagram of the SA-FF module structure of the present invention.

[0056] Figure 6 This is a schematic diagram of the AC module structure of the present invention.

[0057] Figure 7 This is a schematic diagram of the BC module structure of the present invention.

[0058] Figure 8 This is a schematic diagram of the CF module structure of the present invention.

[0059] Figure 9 This is a schematic diagram of the RC module structure of the present invention.

[0060] Figure 10 This is a schematic diagram showing the results of a qualitative comparison between the present invention and current advanced models. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1:

[0063] Embodiment 1 of this invention discloses a boundary-assisted renal tumor segmentation method based on Transformer, such as... Figure 1 As shown, it includes:

[0064] The network structure of the boundary-assisted renal tumor segmentation method based on Transformer, such as Figure 2 As shown.

[0065] Step (1): The backbone features in the endoscopic images are extracted using the PVTv2 backbone extraction network, and then... Figure 3 The Adaptive Context Extraction Module (ACEM) shown performs adaptive extraction of backbone features at different stages.

[0066] The backbone features are four multi-scale pyramid features E1, E2, E3, and E4. Compared with traditional CNN methods, which focus more on local information, PVTv2 shows stronger global information extraction capabilities and better robustness to input interference. The output feature E1 contains rich details such as texture, boundary, and color details, while E2, E3, and E4 have rich semantic information.

[0067] The adaptive context extraction module helps obtain more detailed local feature information. It achieves detail extraction of different receptive domains through dilated convolutions with different dilation rates. Specifically, the adaptive context extraction module adaptively extracts the backbone features:

[0068] The adaptive context extraction module extracts the backbone features (input features). Perform 3x3 dilated convolutions of 1, 3, and 5 to extract multi-scale contextual information from different receptive domains, obtaining features containing multi-scale contextual information. Figure X 1, X2, X3 Furthermore, it effectively reduced background noise interference and preserved local information around the kidney tumor to varying degrees. Multiplying X1, X2, and X3 together yielded O1, as follows:

[0069] O1 = X1⊙X2⊙X3;

[0070] To better capture the surrounding local details of kidney tumors, the extracted multi-scale information was further optimized. An improved spatial attention module of the PVTv2 model, the SA+ module, was proposed. The SA+ module can build a rich contextual relationship model and encode broader contextual information into local features, thereby enhancing its representational power. First, the generated O1 is processed by 1x1 convolution and 1x1 transpose convolution to obtain two new feature maps, O'1 and O”1. And reshape them into Where N = H × W represents the number of pixels;

[0071] Perform matrix multiplication between the two feature maps O'1 and O”1, and perform a softmax operation to compute the spatial attention map S (S∈R). N×N Multiplying this by the feature map O”1 obtained through transpose convolution, we get O2, as follows:

[0072]

[0073]

[0074] in, This is a 1x1 convolution operation; This is a 1x1 transpose convolution operation. The features of each pixel are obtained by multiplying all the other pixels through convolution. All pixels interact with each other, enhancing similar semantic features and improving intra-class compactness and semantic consistency.

[0075] Step (2): Use as follows Figure 4 The Mixed Feature Capture Module (MFCM) shown blends the features obtained from adaptive extraction at different stages with the output features of the previous stage's MFCM, and then uses methods such as... Figure 5 The SA-FF module shown is used for decoding;

[0076] The hybrid feature capture module maps low-resolution features to high-resolution features through upsampling and outputs a mask. It grasps high-order global and local semantic features and can correctly restore them. The hybrid feature capture module has two sub-modules to help with semantic fusion.

[0077] The mixed features are decoded by the SA-FF module, specifically:

[0078] The mixed features are divided into Q, K, and V after passing through three fully connected layers, and then fed into the self-attention-feedforward submodule (SA-FF) for decoding. Utilizing Q, K, and V for computation can better reduce semantic gaps and promote the fusion of high-level and low-level features. Furthermore, through methods such as... Figure 6 The AC module shown efficiently extracts Q and K, and assists the output of the SA-FF module.

[0079] The SA-FF module includes the attention module and forward propagation module inherent in Transformer, as follows:

[0080] O SA-FF =BN(P SRA +V)+MLP(BN(P SRA +V))

[0081] The MLP module is as follows:

[0082] O MLP =ReLU(f c2 (DW(ReLU(f c1 (x)))));

[0083] Where x is the input; f c1 (), f c2 () represents a convolution operation; DW represents a depthwise convolution; ReLU represents an activation function.

[0084] The AC module efficiently extracts Q and K, as follows:

[0085]

[0086] in, This is a 3x3 convolution operation.

[0087] Step (3): The Boundary Assistance Module (BAM) is used to extract the boundary contour features from the backbone features and fuse them with the final output features of the Hybrid Feature Capture Module to obtain the boundary-assisted renal tumor segmentation prediction map.

[0088] The boundary assistance module consists of two parts: one is the boundary extraction submodule BC, such as... Figure 7As shown, the other is the Feature Fusion Submodule (CF), such as... Figure 8 As shown, the BC module extracts the boundary information of the first two layers of the encoder and passes the information to the decoder. It then performs feature fusion with MFC1 through the CF module, thereby increasing the boundary contour details and optimizing the final result.

[0089] A boundary assistance module is used to extract boundary contour features (boundary information of the first two layers of the encoder) from the backbone features, specifically:

[0090] In i Location information is extracted using spatial attention mechanisms, as follows:

[0091]

[0092] in, This is a 3x3 convolution operation; This is a 1x1 convolution operation;

[0093] To obtain more boundary information, O S Input as follows Figure 9 The RC (Residual Convolution) module shown yields intermediate processes f1, f2, and the final result O. BC ,as follows:

[0094]

[0095]

[0096]

[0097] in, This is a 3x3 convolution operation; This is a 1x1 convolution operation.

[0098] The boundary contour features are fused with the final output features from the hybrid feature capture module, as follows:

[0099]

[0100]

[0101]

[0102]

[0103] in, This is a 1x1 convolution operation; This is a 3x3 convolution operation; This is a 1x3 convolution operation; This is a 3x1 convolution operation; This is a 1x5 convolution operation; This is a 5x1 convolution operation.

[0104] It also includes: using the outputs of the hybrid feature capture module and the boundary assistance module as the optimization objective, and defining a loss function by combining the weighted intersection-union loss function (IoU) and the weighted binary cross-entropy loss function (BCE), as follows:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] Among them, y i and y' i These are the ground truth value of the i-th pixel in the n-th stage output and the final output of the frame, respectively. and These are the weighted binary cross-entropy loss and the weighted intersection-union loss, respectively.

[0111] It also includes: using the average dice similarity coefficient mDice, the average intersection-union ratio mIoU, the mean absolute error MAE, and the weighted F-measure. The segmentation performance of renal tumors was quantitatively evaluated using Hausdorff distance (HD), with the following specific metrics:

[0112] The average dice similarity coefficient, mDice, is as follows:

[0113]

[0114] The average crossover ratio mIoU is as follows:

[0115]

[0116] The mean absolute error (MAE) is as follows:

[0117]

[0118] Weighted as follows:

[0119]

[0120]

[0121]

[0122] Hausdorf is the distance from HD as follows:

[0123] H(A,B)=max(h((A,B),(B,A)));

[0124] h(A,B)=max a∈A {min b∈B ||ab||};

[0125] h(B,A)=max b∈ B{min a∈A ||ba||};

[0126] Where TP represents a true positive; FP represents a false positive; FN represents a false negative; n is the number of test images, and y' i and y i represents the prediction and the corresponding ground truth value of the i-th pixel out of a total of n pixels; β is a coefficient with values ​​of 1, 2, and 3; a and b are two point sets used to evaluate the segmentation accuracy of the boundary.

[0127] To evaluate the segmentation performance of the proposed Transformer-based boundary-assisted renal tumor segmentation method, experiments were conducted on Re-TMRS and five polyp segmentation datasets, as follows:

[0128] Re-TMRS: This dataset consists of 2,823 kidney tumor images with resolutions ranging from 720×510 to 1920×1080. The masks in this dataset were annotated and generated under the guidance of relevant endoscopy experts, demonstrating a high level of professionalism.

[0129] Kvasir-SEG: This dataset consists of 1000 polyp images with resolutions ranging from 332×487 to 1920×1072 pixels.

[0130] CVC-ClinicDB: Contains 612 open-access images from 25 colonoscopy videos, with a resolution of 384×288.

[0131] CVC-ColonDB: This dataset contains 380 images with a resolution of 574×500.

[0132] ETIS: This dataset contains 196 images captured from 34 colonoscopy videos, with an image resolution of 1225×996.

[0133] CVC-T: Contains 60 images from 44 colonoscopy sequences, with an image resolution of 574×500.

[0134] When segmenting the Re-TMRS dataset, we randomly selected 2258 images in an 8:2 ratio as the training dataset, and the remaining 565 images as the test dataset. For the polyp segmentation experiment, we used the same training and test datasets as the comparison method. Dataset information is shown in Table 1.

[0135] Table 1 Dataset Information

[0136]

[0137] This invention conducts a performance study on BFT-Net and compares it with current state-of-the-art models, such as... Figure 10 As shown. Advanced methods include Multi-scale-ResNet, CaraNet, DCRNet, HarDNet-MSEG, MSRAformer, Polyp-PVT, and PraNet. From Figure 6 As can be seen from the results, compared with other competing methods, the boundary-assisted renal tumor segmentation method based on Transformer proposed in this invention can segment the renal tumor region more accurately and performs better in several challenging aspects.

[0138] Meanwhile, the five evaluation metrics on this dataset (as shown in Table 2) and the three evaluation metrics on the polyp dataset (as shown in Tables 3 and 4) were analyzed and compared.

[0139] The quantitative results of the proposed Transformer-based boundary-assisted renal tumor segmentation method are statistically compared with six different competing methods on the Re-TMRS dataset, as shown in Table 2.

[0140] Table 2 Comparison of experimental results of different methods on the Re-TMRS kidney tumor dataset

[0141]

[0142]

[0143] The quantitative results of the boundary-assisted renal tumor segmentation method based on Transformer proposed in this invention, compared with seven different competing methods on CVC-ClinicDB and Kvasir, are shown in Table 3.

[0144] Table 3 Comparison of experimental results of different methods on CVC-ClinicDB and Kvasir.

[0145]

[0146] The quantitative results of the proposed Transformer-based boundary-assisted renal tumor segmentation method compared with seven different competing methods on CVC-ColonDB, ETIS, and CVC-T are shown in Table 4.

[0147] Table 4 Comparison of experimental results of different methods on CVC-ColonDB, ETIS, and CVC-T

[0148]

[0149]

[0150] As shown in Table 2, the boundary-assisted renal tumor segmentation method based on Transformer proposed in this invention achieves the best results on the Re-TMRS dataset, with mDice and mIoU reaching 91.1% and 91.8% respectively. The MAE and HD evaluation metrics are also the lowest among all models. The method exhibits the best performance for different values ​​of β, demonstrating its excellent segmentation ability.

[0151] To verify the generalization performance of the proposed Transformer-based boundary-assisted renal tumor segmentation method, tests were conducted on five polyp datasets. Since the polyp training set was selected from CVC-ClinicDB and Kvasir-SEG, the results on these two datasets were evaluated first. Table 3 shows that the proposed Transformer-based boundary-assisted renal tumor segmentation method is optimal across various metrics on both datasets. The mDice of this invention achieves 93.9% and 92.2% in CVC-ClinicDB and Kvasir-SEG, respectively. On the CVC-ClinicDB dataset, the mIoU of this invention outperforms Polyp-PVT and CaraNet by 5.5% and 5.7%, respectively. On the Kvasir-SEG dataset, the mIoU of this invention leads Polyp-PVT and CaraNet by 5.7% and 5.6%, respectively.

[0152] The method was tested on three unseen datasets (CVC-ColonDB, ETIS, and CVC-T). As shown in Table 4, the present invention achieved the best results compared to the seven competing methods. The present invention achieved a significant improvement in mIoU across all three datasets. On CVC-ColonDB, the present invention outperformed Polyp-PVT and CaraNet by 12.7% and 16.5%, respectively. On ETIS, the present invention outperformed Polyp-PVT and CaraNet by 14.3% and 17.7%, respectively. On CVC-T, the present invention outperformed Polyp-PVT and CaraNet by 11.2% and 7.6%, respectively.

[0153] To validate the effectiveness of each component in the model, ablation experiments were conducted on the Adaptive Context Extraction (ACE) module, the Hybrid Feature Capture (MFC) module, and the Boundary Assist (BA) module. The backbone network baseline consisted of PVTv2, and the standard model was composed of "Baseline + ACE + MFC + BA". The effectiveness of different modules was evaluated by removing or modifying different modules from the standard model. "w / o ACE", "w / o MFC", and "w / o BA" represent removing ACE, MFC, or BA from the standard model, respectively. The experimental results are shown in Table 5.

[0154] Table 5 Ablation experimental results on the Re-TMRS dataset

[0155]

[0156] PVTv2 backbone network ablation study: In order to better conduct ablation experiments and intuitively demonstrate the improvement capabilities of each module to the network, a basic "Baseline" was first generated using the PVTv2 backbone network to compare the modules. The experimental results are shown in Table 5.

[0157] Ablation Study of ACE: To quantitatively analyze the effectiveness of the ACE module, a version without the ACE module, "w / o ACE", was trained. All ACE modules were removed, and the network encoder was directly connected to the MFC module. The experimental results are shown in Table 5. Compared with the standard model, the model without ACE showed a sharp decline in performance on the dataset. Compared with the performance of the standard full model, the lack of ACE significantly reduced mIoU, mDice, and... The scores were reduced by 0.4%, 0.6%, 0.3%, 0.5%, and 0.6% respectively, while MAE and HD improved by 0.1% and 9.3% respectively. Therefore, the ACE module can effectively extract contextual feature information.

[0158] MFC Ablation Study: To quantitatively analyze the effectiveness of the MFC module, a version "w / o MFC" without the MFC module was trained. The MFC module was replaced using direct convolution and upsampling. Table 5 shows that, compared to the standard model, the model without MFC significantly improved mIoU, mDice, and... The performance was reduced by 0.3%, 0.5%, 0.2%, 0.3%, and 0.3% respectively, while MAE and HD improved by 0.1% and 3.7% respectively. Therefore, the MFC module can improve the segmentation capability of the network.

[0159] Ablation study of BA: To demonstrate the capabilities of BA, a version "w / oBA" lacking the BA module was trained. Table 5 shows that, compared to the standard model, removing BA leads to lower mIoU, lower mDice, and lower performance. The BA module reduced the margins by 0.2%, 0.4%, and 0.2% respectively, while improving MAE and HD by 0.1% and 8.6% respectively. This demonstrates that the BA module effectively addresses the boundary blurring problem.

[0160] This invention discloses a boundary-assisted renal tumor segmentation method based on Transformer. A PVTv2 backbone extraction network extracts more robust backbone features from endoscopic images, providing more foreground information for subsequent decoding stages, exhibiting stronger global information extraction capabilities and better robustness to input interference. An adaptive context extraction module employs multiple convolutions with different dilation rates to acquire broader features from different receptive domains, effectively extracting local detail information and solving the problem of multi-scale feature adaptation in renal tumors. Furthermore, it can remove background noise interference and reduce misjudgments. A hybrid feature capture module (MFC) addresses the fusion differences between high-level and low-level semantics, narrowing the semantic gap and effectively recovering features. A boundary assistance module (BA) extracts boundary contour features, which are then compared and fused with the output of the MFC to enhance boundary contour details. In summary, this invention effectively extracts local detail information and solves the feature misalignment problem during the fusion of high-level and low-level semantics, as well as the boundary blurring problem of renal tumors.

[0161] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0162] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A boundary-assisted renal tumor segmentation method based on Transformer, characterized in that, include: Step (1): The PVTv2 backbone extraction network is used to extract the backbone features in the endoscopic image, and the adaptive context extraction module is used to adaptively extract the backbone features at different stages. Step (2): The hybrid feature capture module is used to mix the adaptively extracted features from different stages with the output features of the hybrid feature capture module of the previous stage, and then decodes them through the SA-FF module; Step (3): The boundary contour features in the backbone features are extracted using the boundary assistance module and fused with the features finally output by the hybrid feature capture module to obtain the boundary-assisted renal tumor segmentation prediction map; In step (2), the mixed features are decoded by the SA-FF module, specifically as follows: The mixed features are divided into Q, K and V after passing through three fully connected layers and then fed into the SA-FF module for decoding. The AC module efficiently extracts Q and K and assists the output of the SA-FF module. In step (2), the SA-FF module includes the attention module and forward propagation module inherent in Transformer, as follows: in, The output features of the attention module; It is a value vector; for Normalization; Forward propagation module; The MLP module is as follows: in, For the input of the MLP module; , This is a convolution operation; For depthwise convolution; For activation functions; The AC module efficiently extracts Q and K, as follows: in, , 3x3 convolution operation; For element-wise multiplication; Use the Sigmoid activation function; This is a pooling operation; This is the input for the AC module.

2. The boundary-assisted renal tumor segmentation method based on Transformer according to claim 1, characterized in that, In step (1), the backbone feature is a multi-scale pyramid feature.

3. The boundary-assisted renal tumor segmentation method based on Transformer according to claim 1, characterized in that, In step (1), the adaptive context extraction module is used to adaptively extract the backbone features, specifically as follows: The backbone features are subjected to dilated convolution with different dilation rates to obtain feature maps containing multi-scale contextual information, and the feature maps are multiplied together. The SA+ module is used to perform convolution and transpose convolution on the multiplied feature maps respectively to obtain two feature maps, which are then reshaped. Matrix multiplication is performed between the two feature maps, and a softmax operation is performed to compute a spatial attention map, which is then multiplied with the feature map obtained through the transposed convolution.

4. The boundary-assisted renal tumor segmentation method based on Transformer according to claim 1, characterized in that, In step (3), the boundary contour features in the backbone features are extracted using the boundary assistance module, specifically as follows: Location information is extracted using spatial attention mechanisms, as follows: in, This is a 3x3 convolution operation; This is a 1x1 convolution operation; For the first The input feature map is the input to the spatial attention weight main path; This serves as the input to the spatial attention weight branch; Will Input the RC module as follows: in, , This is a 3x3 convolution operation; , , This is a 1x1 convolution operation; This is an element-wise addition.

5. The Transformer-based boundary-assisted renal tumor segmentation method according to claim 1, characterized in that, In step (3), the boundary contour features are fused with the features finally output by the hybrid feature capture module, as follows: in, , This is a 1x1 convolution operation; This is a 3x3 convolution operation; This is a 1x3 convolution operation; This is a 3x1 convolution operation; This is a 1x5 convolution operation; This is a 5x1 convolution operation.

6. The Transformer-based boundary-assisted renal tumor segmentation method according to claim 1, characterized in that, Also includes: Taking the outputs of the hybrid feature capture module and the boundary assistance module as the optimization objective, a loss function is defined by combining the weighted intersection-union loss function and the weighted binary cross-entropy loss function, as follows: in, and These are the ground truth value of the i-th pixel in the n-th stage output and the final output of the frame, respectively. and These are the weighted binary cross-entropy loss and the weighted intersection-union loss, respectively.

Images