Double-branch tumor automatic segmentation network and performance evaluation method thereof

Abstract

The invention discloses a double-branch tumor automatic segmentation network which comprises a semantic segmentation branch and a boundary prior guidance branch. The semantic segmentation branch comprises a double-branch encoder and a single-branch decoder; the double-branch encoder comprises a first encoder block, a global cross-modal interaction module and a shared multi-path down-sampling module; the global cross-modal interaction module is used for extracting global context features of the PET / CT image and performing cross-modal feature interaction; the shared multi-path down-sampling module is used for down-sampling global context features and extracting high-order cross-modal semantic information; the boundary prior guide branch comprises a second encoder block, a down-sampling block, a boundary extraction module and a second decoder block; the boundary extraction module is used for extracting boundary features of the CT image. The invention also discloses a performance evaluation method of the double-branch tumor automatic segmentation network. The technical problem of how to realize accurate and automatic tumor segmentation based on global transmembrane interaction and boundary guidance is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a dual-branch tumor automatic segmentation network and a performance evaluation method thereof. Background Art

[0002] Medical imaging technology plays a key role in the early diagnosis and treatment of cancer. It can help doctors determine the size and location of tumors, as well as the visual relationship between tumors and adjacent tissues. Two commonly used medical imaging technologies for cancer diagnosis are computed tomography (CT) and positron emission tomography (PET). CT provides high-resolution anatomical information, while PET can reflect the biological activity and metabolic function of different tissues and organs. In current clinical diagnosis, tumor segmentation still relies on manual annotation by doctors, which is time-consuming and labor-intensive, and is easily affected by differences in doctors' professional knowledge and opinions. Therefore, computer-aided diagnosis technology has been widely used in the field of tumor segmentation.

[0003] In computer-aided diagnosis (CAD), automatic tumor segmentation methods based on multimodal scans have become a research hotspot. A key challenge in multimodal tumor segmentation lies in effectively leveraging complementary features. Existing methods typically extract multi-stage features from PET and CT images separately and then integrate the feature representations of the two modalities through complexly designed modules and network architectures. However, these methods rely on implicitly learning the correlations and interactions between CT and PET images. Multimodal feature representations have not been fully explored. PET / CT images alone are not perfect, and they need to leverage beneficial features from the corresponding modality images to achieve accurate tumor segmentation. Furthermore, extracting global dependencies between image features is a key step in achieving accurate tumor segmentation. Common methods for obtaining global contextual information introduce high computational complexity and relatively low efficiency. Automatic tumor segmentation plays a crucial role in tumor diagnosis and treatment planning. CT and PET provide complementary medical information that is widely used. However, existing methods ignore the bilateral cross-modal interactions of global features during feature extraction and fail to fully utilize multi-stage tumor boundary features. To address these issues, this application proposes a dual-branch automatic tumor segmentation network and its performance evaluation method. Summary of the Invention

[0004] The main purpose of this invention is to propose a dual-branch tumor automatic segmentation network and its performance evaluation method, aiming to solve the technical problem of how to achieve accurate automatic tumor segmentation based on global transmembrane state interaction and boundary guidance.

[0005] To achieve the above objectives, the present invention provides a dual-branch tumor automatic segmentation network, wherein the dual-branch tumor automatic segmentation network comprises:

[0006] Semantic segmentation branch and boundary prior guidance branch;

[0007] The semantic segmentation branch includes a dual-branch encoder and a single-branch decoder;

[0008] The dual-branch encoder includes a first encoder block, a global cross-modal interaction module, and a shared multi-path downsampling module; the global cross-modal interaction module is used to extract global context features of PET / CT images and perform cross-modal feature interaction; the shared multi-path downsampling module is used to downsample the global context features and extract high-order cross-modal semantic information; the single-branch decoder includes a first decoder block and a transposed convolutional layer;

[0009] The boundary prior guidance branch includes a second encoder block, a downsampling block, a boundary extraction module and a second decoder block; the boundary extraction module is used to extract boundary features of the CT image.

[0010] In one of the preferred solutions, the global cross-modal interaction module includes a CT branch;

[0011] The CT branch includes a first convolutional layer, a first 2D fast Fourier transform, a first global filter, a first Fourier channel attention block and a first 2D fast inverse Fourier transform; the CT branch is used to extract global context features of the CT image based on the CT image and the boundary features output by the boundary extraction module.

[0012] In one of the preferred solutions, the global cross-modal interaction module includes a PET branch;

[0013] The PET branch includes a second convolutional layer, a second 2D fast Fourier transform, a second global filter, a second Fourier channel attention block and a second 2D fast inverse Fourier transform; the PET branch is used to extract global context features of the PET image based on the PET image.

[0014] In one preferred solution, the global cross-modal interaction module extracts global context features of PET / CT images, specifically:

[0015] Input the PET image into the PET branch, and the CT image and boundary features into the CT branch;

[0016] Perform lossless conversion between frequency domain and spatial domain;

[0017] Extracting global context features from PET and CT images, performing element-by-element cross-merging on the global context features, and sharing parameter weights to obtain feature representations of images of different modalities;

[0018] According to the feature representation of different modal images, the channel features in the frequency domain are obtained based on FCA, and the weight factor of each channel is obtained through the activation function;

[0019] The feature information fusion of spatial domain and frequency domain is achieved through residual connection.

[0020] In one of the preferred solutions, the shared multi-path downsampling module includes a third convolutional layer, a maximum pooling layer and a channel attention block.

[0021] In one of the preferred solutions, the boundary extraction module includes a Sobel operator, a global filter layer, a convolution layer, a batch processing layer and ReLU activation.

[0022] In one preferred embodiment, the Sobel operator includes a horizontal boundary detection operator SobelX and a vertical boundary detection operator SobelY;

[0023] The horizontal boundary detection operator SobelX is used to obtain the horizontal gradient map F of the CT image. bx ;

[0024] The vertical boundary detection operator SobelY is used to obtain the vertical gradient map F of the CT image by .

[0025] In one preferred embodiment, the boundary features of the CT image are:

[0026]

[0027] Among them, F bxy is the boundary feature of the CT image, Sigmoid() is the activation function, F bx is the horizontal gradient map of the CT image, F by is the vertical gradient map of the CT image.

[0028] A performance evaluation method for a dual-branch tumor automatic segmentation network comprises the following steps:

[0029] Set up evaluation datasets and evaluation metrics;

[0030] Preprocess CT / PET scans and use data augmentation to expand the evaluation dataset;

[0031] Based on the evaluation data set, an ablation experiment is performed on each branch module in the dual-branch tumor automatic segmentation network to determine the effectiveness of each branch module in the dual-branch tumor automatic segmentation network.

[0032] In one of the preferred solutions, the evaluation indicators include dice coefficient, intersection-over-union ratio, positive predictive value, recall rate and Hausdorff distance.

[0033] In the above technical solution of the present invention, the dual-branch tumor automatic segmentation network includes: a semantic segmentation branch and a boundary prior guidance branch; the semantic segmentation branch includes a dual-branch encoder and a single-branch decoder; the dual-branch encoder includes a first encoder block, a global cross-modal interaction module and a shared multi-path downsampling module; the global cross-modal interaction module is used to extract global context features of PET / CT images and perform cross-modal feature interaction; the shared multi-path downsampling module is used to downsample the global context features and extract high-order cross-modal semantic information; the single-branch decoder includes a first decoder block and a transposed convolutional layer; the boundary prior guidance branch includes a second encoder block, a downsampling block, a boundary extraction module and a second decoder block; the boundary extraction module is used to extract boundary features of CT images. The present invention solves the technical problem of how to achieve accurate automatic tumor segmentation based on global cross-modal interaction and boundary guidance.

[0034] In the present invention, the global transmodal interaction module extracts key global contextual features from the frequency domain of PET / CT images through a global filter, and integrates them into the corresponding modal branches through cross-fusion to promote cross-modal feature interaction and help learn complementary cross-modal feature representations without affecting the high-level feature extraction in a single modality branch.

[0035] In the present invention, the shared multi-path downsampling module enhances the network's ability to utilize complementary features in the PET / CT modality through weight sharing, thereby improving tumor segmentation performance. In addition, the shared multi-path downsampling module integrates multiple downsampling methods to reduce feature loss during the downsampling process.

[0036] In the present invention, the boundary extraction module extracts potential boundary features from multiple stages of the CT image, and assists the semantic segmentation branch to improve the accuracy of tumor boundary segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0038] Figure 1 This is a first schematic diagram of a dual-branch tumor automatic segmentation network according to an embodiment of the present invention;

[0039] Figure 2 This is a second schematic diagram of a dual-branch tumor automatic segmentation network according to an embodiment of the present invention;

[0040] Figure 3 Schematic diagram of a global cross-modal interaction module according to an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of a shared multi-path downsampling module according to an embodiment of the present invention;

[0042] Figure 5 Schematic diagram of a boundary extraction module according to an embodiment of the present invention;

[0043] Figure 6 Schematic diagram of qualitative segmentation results of four test cases on the STS dataset according to an embodiment of the present invention;

[0044] Figure 7 Schematic diagram of qualitative segmentation results of four test cases in the Hecktor 2022 dataset according to an embodiment of the present invention;

[0045] Figure 8 Schematic diagram of tumor boundary segmentation results on the STS dataset and the Hecktor 2022 dataset according to an embodiment of the present invention;

[0046] Figure 9 Schematic diagram of a comparative architecture of a global cross-modal interaction module according to an embodiment of the present invention;

[0047] Figure 10 Schematic diagram of a performance evaluation method for a dual-branch tumor automatic segmentation network according to an embodiment of the present invention;

[0048] Figure 11 This is a comparison of PET and CT cross-sectional images of patients with head and neck tumors and soft tissue sarcomas in an embodiment of the present invention;

[0049] Figure 12 Schematic diagram of three different fusion strategies in the multimodal segmentation method according to an embodiment of the present invention.

[0050] The realization of the objectives, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0052] In addition, the terms "first," "second," and so on, used in this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include at least one of these features.

[0053] Moreover, the technical solutions between the various embodiments of the present invention may be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0054] See also Figure 1 According to one aspect of the present invention, a dual-branch tumor automatic segmentation network is provided, wherein the dual-branch tumor automatic segmentation network comprises:

[0055] Semantic segmentation branch and boundary prior guidance branch;

[0056] The semantic segmentation branch includes a dual-branch encoder and a single-branch decoder;

[0057] The dual-branch encoder includes a first encoder block, a global cross-modal interaction module, and a shared multi-path downsampling module; the global cross-modal interaction module is used to extract global context features of PET / CT images and perform cross-modal feature interaction; the shared multi-path downsampling module is used to downsample the global context features and extract high-order cross-modal semantic information; the single-branch decoder includes a first decoder block and a transposed convolutional layer;

[0058] The boundary prior guidance branch includes a second encoder block, a downsampling block, a boundary extraction module and a second decoder block; the boundary extraction module is used to extract boundary features of the CT image.

[0059] Specifically, in this embodiment, see Figure 2The dual-branch tumor automatic segmentation network includes a semantic segmentation branch and a boundary prior guidance branch. The semantic segmentation branch is used to obtain a visual representation of the tumor from multimodal images. The boundary prior guidance branch is used to extract boundary feature information from CT images to assist the semantic segmentation branch in tumor boundary segmentation. The semantic segmentation branch extracts a visual representation of the tumor from multimodal images. The boundary prior guidance branch focuses on multi-stage boundary feature extraction, assisting the semantic segmentation branch in performing accurate tumor boundary segmentation and reducing the uncertainty of the model in boundary areas. The semantic segmentation branch consists of two key components: a global cross-modal interaction module and a shared multi-path downsampling module. The global cross-modal interaction module helps learn complementary cross-modal feature representations without negatively impacting high-level feature extraction in a single modality branch. The global cross-modal interaction module uses a global filter to extract key global contextual features from the frequency domain of PET / CT images and integrates them into the corresponding modality branch through cross-fusion to promote cross-modal feature interaction. The shared multi-path downsampling module enhances the model's ability to utilize complementary features in PET / CT modalities through weight sharing, thereby improving tumor segmentation performance. In addition, the shared multi-path downsampling module integrates multiple downsampling methods to reduce feature loss in the downsampling process; in the boundary prior guided branch, the boundary extraction module can extract potential boundary features from the CT image. By applying the boundary extraction module at different stages, the boundary prior guided branch can obtain multi-level boundary features; among them, the semantic segmentation branch (Semantic Segmentation Branch, hereinafter referred to as SSB), the global cross-modal interaction module (Global Cross-modal Interaction Module, hereinafter referred to as GCIM), the shared multi-path downsampling module (Shared Multi-path Downsampling Module, hereinafter referred to as SMDM), the boundary prior guided branch (Boundary Prior-guided Branch, hereinafter referred to as BPGB), the boundary extraction module (Boundary Extraction Module, hereinafter referred to as BEM); Seg Loss and Edge Loss both represent loss functions.

[0060] Specifically, the semantic segmentation branch is based on the UNet architecture and includes a dual-branch encoder and a single-branch decoder; the dual-branch encoder is used to extract multimodal feature representations, and the single-branch decoder is used to obtain reconstructed segmentation results; the global cross-modal interaction module extracts global context features from the frequency domain of PET / CT images to promote cross-modal interaction of global features; the shared multi-path downsampling module learns cross-modal semantic information to reduce the loss of feature information during the downsampling process; the first encoder block and the first decoder block both contain two sets of convolutional layers, batch normalization layers and ReLU activation functions; jump connections are established between encoder blocks and decoder blocks with the same dimension to retain the original feature information; the goal of the semantic segmentation branch is to learn multimodal complementary image features to improve the accuracy of tumor segmentation.

[0061] Specifically, in this embodiment, see Figure 3 , the global cross-modal interaction module extracts key global context features from the frequency domain of PET / CT images and merges them into another branch through cross fusion, the global cross-modal interaction module promotes complementary feature interaction without negatively affecting the extraction of high-level features by a single branch; the global cross-modal interaction module includes a CT branch and a PET branch; the CT branch takes the CT image and the boundary features extracted by the boundary extraction module as input, the PET branch takes the PET image as input, and both the CT branch and the PET branch are designed to extract high-level semantic features from the CT image and the PET image; the CT branch and the PET branch are composed of the same structure; The CT branch includes a first convolutional layer, a first 2D fast Fourier transform (FFT), a first global filter, a first Fourier channel attention block and a first 2D fast inverse Fourier transform (IFFT); the CT branch is used to extract global context features of the CT image based on the CT image and the boundary features output by the boundary extraction module; the PET branch includes a second convolutional layer, a second 2D fast Fourier transform (FFT), a second global filter, a second Fourier channel attention (FCA) block and a second 2D fast inverse Fourier transform (IFFT); the PET branch is used to extract global context features of the PET image based on the PET image.

[0062] Specifically, in this embodiment, the global cross-modal interaction module extracts global context features of PET / CT images, specifically:

[0063] Input the PET image into the PET branch, and the CT image and boundary features into the CT branch;

[0064] Lossless conversion between frequency domain and spatial domain is performed through 2D fast Fourier transform; specifically:

[0065]

[0066] Among them, X ct is the input of the CT branch, X pet is the input of the PET branch, f[] represents the 2D fast Fourier transform, F ct Enter X for the CT branch ct The spectrum, F pet Enter X for the PET branch pet spectrum;

[0067] The global context features of PET and CT images are extracted, and the global context features are cross-merged element by element, and parameter weights are shared to obtain feature representations of images of different modalities. To promote cross-modal interaction of global features, global filters are used to extract global context features from PET and CT images respectively, and cross-merged into their respective branches through element-by-element addition. In each branch, the global filters share parameter weights, enabling them to learn feature representations of images of different modalities. The feature representations of the images of different modalities are:

[0068]

[0069] in, represents element-wise addition, ⊙ is the Hadamard product, which represents the new matrix obtained by multiplying the corresponding elements in the two matrices, K is the shared parameter weight, F' ct is the feature representation of the CT branch, F' pet It is the characteristic representation of PET branch;

[0070] Based on the feature representation of different modal images, FCA is used to obtain channel features in the frequency domain, and the weight factor of each channel is obtained through the activation function. To capture channel feature information from the frequency tensor in the frequency domain, FCA is used to obtain channel features in the frequency domain. FCA uses global average pooling to obtain a global understanding of each channel, and the weight factor of each channel is obtained through the sigmoid function. FCA enhances the network's global channel understanding ability and emphasizes learning feature maps related to tumor regions. Specifically:

[0071]

[0072] Among them, F” ct is the channel characteristic of the CT branch, F” pet Pooling of channel characteristics for PET branches global () is the global average pooling operation;

[0073] The feature information fusion of spatial domain and frequency domain is realized through residual connection; specifically:

[0074]

[0075] Among them, F' ct It is the feature information of CT branch after fusion of spatial domain and frequency domain. pet It is the feature information of the PET branch after the fusion of spatial domain and frequency domain; is element-wise multiplication.

[0076] Specifically, in this embodiment, see Figure 4 To help the encoder improve the effectiveness of extracting complementary features from PET / CT modes and minimize feature loss during the downsampling process, the present invention proposes a shared multi-path downsampling module; during the network training process, the weight of each shared multi-path downsampling module is updated twice within one iteration, so that the shared multi-path downsampling module can learn features from different modalities, reducing the complexity of the model. Therefore, when F"' ct Obtain high-order cross-modal semantic information F” through shared multi-path downsampling modules ct When, F ct will contain the modality features of the relevant PET images; similarly, when F' pet Get F” by sharing multi-path downsampling modules pet When, F pet The modal features of the relevant CT images are included; the shared multi-path downsampling module includes a third convolutional layer, a maximum pooling layer and a channel attention block; the third convolutional layer uses a 2×2 convolutional layer to preserve spatial information, thereby enhancing the multi-scale learning ability of the model; the maximum pooling layer extracts significant features from the PET / CT images; in order to reduce the loss of discriminant features, the present invention adopts an element-by-element addition method combined with two downsampling methods; subsequently, the enhanced channel attention block is used to capture global channel information and selectively emphasize the learning feature map related to the tumor area in the input image.

[0077] Specifically, in this embodiment, see Figure 5In order to make full use of multi-stage potential boundary features to enhance the tumor boundary segmentation ability of the semantic segmentation branch, the present invention introduces a boundary prior guidance branch. Since CT images can reflect the physical structure differences between different tissues and provide clear boundary information, the present invention uses CT images as input. The boundary prior guidance branch is constructed based on the UNet framework and consists of a second encoder block, a downsampling block, a boundary extraction module and a second decoder block. In order to extract potential boundary feature information from CT images, the present invention proposes a boundary element method. The boundary extraction module includes a Sobel operator, a global filter layer, a convolution layer, a batch layer and a ReLU activation. The Sobel operator is used to extract boundary information from the CT image. The Sobel operator includes a horizontal boundary detection operator SobelX and a vertical boundary detection operator SobelY. The horizontal boundary detection operator SobelX is used to obtain the horizontal gradient map F of the CT image. bx The vertical boundary detection operator SobelY is used to obtain the vertical gradient map F of the CT image by ; The final boundary feature is obtained by the sigmoid function; the boundary feature of the CT image is:

[0078]

[0079] Among them, F bxy is the boundary feature of the CT image, Sigmoid() is the activation function, F bx is the horizontal gradient map of the CT image, F by is the vertical gradient map of the CT image; X ct is the input CT image;

[0080] The horizontal boundary detection operator SobelX is The vertical boundary detection operator SobelY is

[0081]

[0082]

[0083] In order to bxy To further extract tumor boundary features, the present invention extracts global boundary context features through a global filter, and uses 3×3 convolution to extract local boundary features. 1×1 convolution reduces the feature map to a single channel, thereby obtaining a final boundary feature map. The final boundary feature map is passed to the next stage to obtain a tumor boundary feature map of the or stage; the final boundary feature map is:

[0084] F' bxy =Conv 1×1 (Conv 3×3 (Filterglobal (F bxy )))

[0085] Among them, F' bxy is the final boundary feature map, Filter global () is the global filtering operation.

[0086] Specifically, in this embodiment, in the dual-branch tumor automatic segmentation network, the total loss function L total Including L SSB and L BPGB The weighted sum of SSB is the loss function of the semantic segmentation branch, L BPGB is the loss function of the boundary prior guided branch. For the semantic segmentation branch, the present invention adopts a combination of BCE loss and Dice loss to optimize the network weights, and the boundary prior guided branch adopts Dice loss to optimize the network weights; the total loss function is:

[0087] L total =L SSB +λL BPGB

[0088]

[0089]

[0090] Where λ is the boundary loss L BPGB The weight factor of L is λ=0.1 in the present invention, which is not specifically limited in the present invention and can be set according to needs; α is L SSB The weight factor, S i and G i Represent the segmentation prediction result and the true result at pixel i, N is the total number of pixels, G sr and G gt They represent the segmentation prediction result set and the true result set respectively, and || is the number of pixels in the set.

[0091] According to another aspect of the present invention, a performance evaluation method for a dual-branch tumor automatic segmentation network is provided, comprising the following steps:

[0092] Set up evaluation datasets and evaluation metrics;

[0093] Preprocess CT / PET scans and use data augmentation to expand the evaluation dataset;

[0094] Based on the evaluation data set, an ablation experiment is performed on each branch module in the dual-branch tumor automatic segmentation network to determine the effectiveness of each branch module in the dual-branch tumor automatic segmentation network.

[0095] Specifically, in this embodiment, in the experimental stage, the present invention uses various publicly accessible datasets to evaluate the dual-branch tumor automatic segmentation network, including the soft tissue sarcoma (Soft Tissue Tarcoma, hereinafter referred to as STS) dataset and the head and neck tumor (Hecktor) 2022 dataset; in addition, in the training stage, the present invention applies k-fold cross-validation to each dataset separately to maximize data utilization and reduce overfitting; the STS dataset contains PET / CT scan data from 51 patients diagnosed with STS in the limbs, and these data come from different scanners; the CT scan in-plane resolution in the STS dataset is 512×512, and the slice range is [98,313]. The in-plane pixel pitch is 0.98 mm × 0.98 mm, and the slice thickness is 3.27 mm. The in-plane resolution of PET scans is 128 × 128, which is comparable to the number of slices in CT scans. The in-plane pixel pitch of PET slices is 3.91 mm × 3.91 mm, and the slice thickness is 3.27 mm. To evaluate the performance of the network, we selected 36 cases (1826 slices) from the STS dataset as a cross-validation dataset, and the remaining 15 cases (685 slices) as a test set. The Hecktor 2022 dataset includes PET / CT scans of 524 patients diagnosed with oropharyngeal head and neck cancer. These scans were collected from nine different centers using different scanning machines and protocols. In the Hecktor 2022 dataset, the in-plane resolution of CT scans is 512 × 512, and the slices range from 91 to 736. The pixel pitch of CT slices is 0.98 to 1.36 mm, and the slice thickness is 3.27 mm. The in-plane resolution of PET scans is 128 × 128, which is the same number of slices as CT scans. The intra-slice pixel spacing is 2.73-5.47 mm, and the slice thickness is 3.27 mm. To evaluate the performance of the trained network, we selected 370 cases (17,245 slices in total) from the Hecktor2022 dataset as a cross-validation dataset, and the remaining 154 cases (7,487 slices in total) as a test set.

[0096] Specifically, in this embodiment, CT / PET scans are preprocessed; first, the Hounsfield units (HU) of the CT scans are truncated in the range of [-260, 260]; then, bilinear interpolation is used to resample the CT scans to 256×256; then, the PET scans are truncated in the range of [0, 5] using the standard uptake value (SUV); using the resampled CT scans as a spatial alignment reference, the PET scans are resampled to 256×256 using bilinear interpolation; finally, the CT and PET scans are normalized using zero-mean normalization; during the evaluation process, the PET / CT volumes are converted into two-dimensional images. In order to alleviate the size limitation of the STS dataset, data augmentation techniques are used to expand the training dataset, including random rotation, horizontal flipping, vertical flipping, and contrast processing.

[0097] Specifically, in this embodiment, five evaluation indicators are used to evaluate the segmentation performance of the dual-branch tumor automatic segmentation network, including: Dice coefficient (Dice), intersection over union (IOU), positive predictive value (PPV), recall rate, and Hausdorff distance (HD). The calculation formula of each evaluation indicator is as follows:

[0098]

[0099]

[0100]

[0101]

[0102] HD(X,Y)=max x∈X {min y∈Y {d(x,y)}}

[0103] Among them, Dice is the dice coefficient, IOU is the intersection-over-union ratio, PPV is the positive predictive value, Recall is the recall rate, HD is the Hausdorff distance, TP, FP, and FN are commonly used indicators for evaluating the performance of classification models, which are true positive examples, false positive examples, and false negative examples, respectively.

[0104] Specifically, in this embodiment, the experimental details are processed, and all networks in the experiment are implemented using the existing Python3.8 and PyTorch 2.0.0 framework. The present invention does not provide specific instructions, and the conventional method can be used; the training is carried out on the NVIDIA RTX 3090 GPU, and the optimizer used in the experiment is stochastic gradient descent with 0.9 momentum. For hyperparameters, the learning rate is set to 0.0001 and the batch size is set to 16; the experiment is evaluated using 5-fold cross validation; during the training process, the early stopping method is applied, that is, if the loss function of the validation set does not decrease for 10 consecutive cycles, the model training is stopped.

[0105] Specifically, in this embodiment, the present invention conducted separate ablation experiments on the STS dataset and the Hecktor 2022 dataset to evaluate the segmentation performance of the dual-branch tumor automatic segmentation network; the present invention conducted separate ablation experiments to verify the contribution of each component in the dual-branch tumor automatic segmentation network; first, the present invention constructed a baseline model based on the UNet architecture; the baseline model consists of a dual-branch encoder and a single-branch decoder; in order to evaluate the effectiveness of different components in the dual-branch tumor automatic segmentation network, the present invention conducted experiments with seven different settings on the two datasets. Tables 1 and 2 show the quantitative results of ablation experiments conducted on the STS dataset and the Hecktor 2022 dataset, respectively. Compared with the baseline network, the global cross-modal interaction module improves the Dice score by 2.37%, the IOU score by 2.4%, and reduces the Hausdorff distance (HD) by 7.01mm on the STS dataset; in addition, the global cross-modal interaction module improves the Dice score by 2.8%, the IOU score by 3.51% on the Hecktor 2022 dataset, and the Hausdorff distance (HD) increases slightly to 0.32mm; the above results show that the global cross-modal interaction module enhances the network's utilization of multimodal complementary feature information, thereby having a positive impact on the segmentation performance. Compared with the baseline model, the shared multi-path downsampling module improved the Dice score by 1.98%, the IOU score by 1.68%, and the recall rate by 1.91% on the STS dataset; in addition, the shared multi-path downsampling module improved the Dice score by 1.21%, the IOU score by 2.95%, and the recall rate by 2.67% on the Hecktor 2022 dataset; the above results show that the shared multi-path downsampling module enhances the tumor segmentation ability of the model by learning the complementary features of PET / CT modalities within the same spatial distribution. Compared with the baseline model, the boundary prior guided branch improves the Dice score by 2.91%, the IOU score by 1.33%, and reduces the Hausdorff distance by 11.03mm on the STS test dataset; in addition, the boundary prior guided branch improves the Dice score by 2.01%, the IOU score by 3.31%, and reduces the Hausdorff distance by 0.9mm on the Hecktor 2022 test dataset; these findings indicate that the auxiliary boundary features at different stages provided by the boundary prior guided branch reduce the uncertainty of the model in boundary segmentation, a hypothesis supported by the results obtained from the boundary-sensitive Hausdorff distance (HD) metric.

[0106] In addition, the results in Tables 1 and 2 show that when the global cross-modal interaction module is combined with the shared path downsampling module and the boundary prior guided branch, the segmentation performance of the network is significantly improved, which confirms the effectiveness of the network proposed in this paper.

[0107] Table 1 Ablation experiments on the STS dataset (average value)

[0108]

[0109] Table 2 Ablation experiment on Hecktor 2022 dataset (average value)

[0110]

[0111] In order to analyze the impact of different loss functions on the network segmentation performance, the present invention uses the Hecktor 2022 dataset to experiment with various common loss functions; the compared loss functions include BCE loss, Dice loss, Focal loss and the weighted BCE+Dice loss used in the present invention; the results are shown in Table 3. Among the single loss functions, Dice loss shows superior segmentation performance and exceeds other loss functions in all indicators. The weighted BCE+Dice loss of the present invention combines the advantages of Dice loss and BCE loss by assigning different weights, thereby producing more competitive segmentation performance. It is 1.27% higher than Dice loss in Dice score, 0.98% higher in recall rate, and reduces the HD distance by 1.84mm; in addition, based on the comprehensive experimental results of the present invention, the optimal weight distribution of Dice loss and BCE loss is determined to be 0.8 and 0.2, respectively.

[0112] Table 3 Comparison of loss functions

[0113]

[0114] Specifically, in this example, the present invention compares the dual-branch tumor automatic segmentation network and the state-of-the-art methods on the STS dataset and the Hecktor 2022 dataset. To ensure a fair comparison, the present invention uses consistent preprocessing methods and experimental details for all methods. The compared methods include unimodal UNet, dense UNet, deep dilated segmentation network (DiSegNet), multimodal spatial attention module method (MSAM), collaborative learning feature fusion method (CoFeatureModel), and UNet-based visual transformer (TransUNet).

[0115] Specifically, in this embodiment, Table 4 shows the quantitative segmentation results of all methods on the STS dataset. Table 4 shows that UNet++ has the lowest Dice and Recall scores, which are 0.7560 and 0.7603 respectively; DiSegNet has the lowest IOU, PPV and HD scores, which are 0.6223, 0.7180 and 20.7684 mm respectively; in contrast, DGCBG-Net performs well in Dice, PPV, Recall and HD scores, reaching 0.8033, 0.8114, 0.8428 and 12.0168 mm respectively; in addition, its IOU score is 0.6764, second only to TransUNet, see Figure 6 The visual segmentation results of the STS dataset are shown, among which DGCBG-Net is a dual-branch tumor automatic segmentation network; it is obvious that the segmented image predicted by the dual-branch tumor automatic segmentation network has the largest true positive area and the smallest false negative area. This significant improvement can be attributed to the enhanced cross-modal interaction of the global cross-modal interaction module and the shared multi-path downsampling module, which allow more comprehensive utilization of multimodal feature representation; in addition, the boundary prior guided branch of the present invention improves the accuracy of the network in segmenting tumor boundary areas and significantly reduces the false positive area; quantitative and qualitative results show that the dual-branch tumor automatic segmentation network of the present invention achieves better segmentation effect under the same conditions; among them, Baseline represents the baseline model, Unet++ represents dense UNet, DiSegNet represents deep dilated segmentation network, CofeatureModel represents collaborative learning feature fusion method, MSAM represents multimodal spatial attention module method, and TransUNet represents UNet-based visual converter.

[0116] Table 4 Quantitative evaluation results of each method on the STS dataset (average value)

[0117]

[0118] Specifically, in this embodiment, Table 5 shows the quantitative segmentation results of various methods on the Hecktor 2022 dataset. It is obvious that UNet++ has the lowest Dice score of 0.7645, and CoFeatureModel has the lowest IOU and Recall scores of 0.6362 and 0.7101 respectively; MSAM has the lowest PPV score of 0.8686mm, and TransUNet has the lowest HD score of 12.7131mm; DGCBG-Net performs well in Dice, IOU and HD scores, reaching the highest value (Dice: 0.7929, IOU: 0.7018, HD: 8.7219mm), and it ranks second in PPV and Recall scores (PPV is 0.8816, Recall is 0.7566); the dual-branch tumor automatic segmentation network always presents the largest true positive area, the smallest false positive area and the smallest false negative area. In addition, Figure 7 In the bottom case, our method correctly predicts a small tumor region surrounding a large tumor region. In contrast, TransUnet predicts an incorrect location, and the other methods fail to predict this tumor region. This observation emphasizes the positive impact of the global contextual features extracted by the global cross-modal interaction module on the accuracy of small tumor region segmentation. Furthermore, cross-modal information exchange effectively reduces the likelihood of incorrect segmentation by the model.

[0119] Table 5 Quantitative evaluation results of each method on the Hecktor 2022 dataset (mean)

[0120]

[0121] Specifically, in this embodiment, to comprehensively analyze the differences between the dual-branch tumor automatic segmentation network and the state-of-the-art methods, the present invention selected two recent methods for segmentation performance comparison. The prior art proposed C2BA-UNet, which contains two key components: a multi-group boundary perception module that effectively captures the feature information of the tumor boundary area by combining gradient maps, uncertainty maps, and level maps; and a context coordination module that effectively integrates multi-scale information using an attention mechanism. The prior art proposed ASE-Net, which includes two key components: a pseudo-enhanced CT image method based on metabolic intensity to enhance the location uniqueness of high- and low-metabolism regions, and an adaptive scale attention supervision module that learns tumor features at different scales. Both works conducted extensive experiments on the STS dataset and achieved competitive segmentation performance. However, since these methods do not provide open source code, the present invention is unable to replicate their experimental procedures. Therefore, we follow the experimental settings outlined in these two papers and retrain our dual-branch tumor automatic segmentation network, where the input image resolution of C2BA-UNet is 128x128 and the input image resolution of ASE-Net is 256x256. We compare our results with those of C2BA-UNet and ASE-Net on four performance metrics: Dice, IOU, Recall, and PPV. As shown in Table 6, our method outperforms the state-of-the-art in most metrics under both resolution settings. Specifically, at a resolution of 128x128, the method of the present invention surpassed C2BA-UNet with a Dice score of 1.62% and an IOU score of 0.82%. At a resolution of 256x256, the method of the present invention surpassed ASE-Net with a Dice score of 1.77%, an IOU score of 2.12%, a Recall score of 1.59%, and a PPV score of 3.35%. These results demonstrate that the dual-branch tumor automatic segmentation network of the present invention is able to achieve excellent performance in tumor segmentation tasks regardless of the resolution of the input image. In addition, compared with the most advanced methods, the method of the present invention showed significant advantages in multiple key indicators, which further demonstrated its effectiveness and practicality. These experimental results strongly demonstrate that the dual-branch tumor automatic segmentation network outperforms the SOTA method in segmentation performance.

[0122] Table 6 Quantitative evaluation results of SOTA methods on the STS dataset (mean)

[0123]

[0124] Specifically, in this embodiment, in order to further analyze the segmentation performance of the dual-branch tumor automatic segmentation network in the tumor boundary area, the present invention evaluates the boundary segmentation results of all methods on the STS dataset and the Hecktor 2022 dataset respectively. The visualized boundary segmentation results are shown in Figure 8 , the upper two rows are the boundary segmentation results of the Hecktor 2022 dataset, and the lower two rows are the boundary segmentation results of the STS dataset. On the Hecktor 2022 dataset, all methods did not segment the edges sufficiently, and the compared methods had poor segmentation effects in some smaller or irregularly shaped tumor areas; on the contrary, the dual-branch tumor automatic segmentation network of the present invention achieved better segmentation effects in these areas by utilizing the multi-scale boundary feature information provided by the boundary prior guided branch; on the STS dataset, all methods showed over-segmentation, especially in areas with irregular tumor edges. In comparison, these methods found it difficult to achieve accurate segmentation. Due to the boundary auxiliary information provided by the boundary prior guided branch, the method of the present invention is closer to the actual situation in these areas.

[0125] Specifically, in this example, Table 7 summarizes the parameter counts and computational resource requirements of all methods. Clearly, the dual-branch tumor automatic segmentation network involves fewer parameters and computational resources than most methods. In particular, it has significantly fewer parameters than TransUNet. Although it does require more parameters and computational resources than the smallest CoFeatureModel, the dual-branch tumor automatic segmentation network achieves an effective balance between segmentation performance and computational cost.

[0126] Table 7 Comparison of model size and calculation represented by Params and Flops

[0127]

[0128] Specifically, in this embodiment, automatic tumor segmentation is a key step in computer-aided disease diagnosis and treatment planning. The present invention proposes a dual-branch tumor automatic segmentation network (DGCBG-Net) based on global cross-modal interaction and boundary guidance for tumor segmentation. The dual-branch tumor automatic segmentation network includes three key components: GCIM, SMDM, and BPGB. The present invention conducts extensive ablation and comparison experiments on two public datasets; the experimental results are shown in Tables 1-7 and Figure 6-Figure 9 As shown in Figure 2, the effectiveness of the three key components and the advantages of DGCBGNet in automatic tumor segmentation are demonstrated. In addition, the present invention conducts additional experiments to analyze the rationality of the proposed key modules. The present invention designs a set of comparative experiments for the global cross-modal interaction module, transforming the cross-modal interaction method into an early fusion method, such as Figure 9As shown. The experimental results on the STS dataset are shown in Table 8. When the comparison architecture is used as the cross-modal interaction method, the Dice score, IOU score and Recall score are all lower than the Baseline results. The present invention infers that the reason for this situation is the feature splicing along the channel dimension, which causes the model to receive a large number of redundant features. These redundant features cause the model to allocate more attention to the background pixel area rather than the foreground pixel area, thereby affecting the segmentation performance of the network. In contrast, GCIM ensures the comprehensive utilization of global feature representations from different modalities by integrating global filters and bilateral cross-modal feature interactions. The experimental results in Table 8 further verify the effectiveness of GCIM.

[0129] Table 8 Comparative experimental results of GCIM on the STS dataset

[0130]

[0131] This paper analyzes the impact of weight sharing in a shared multipath downsampling module (SMDM) by introducing a multipath downsampling module (MDM) as a comparative reference. The experimental results, shown in Table 9, demonstrate that SMDM outperforms MDM in Dice score, IOU score, PPV score, recall score, and HD score, improving by 0.59%, 3.31%, 4.7%, 1.22%, and 0.8514 mm, respectively, while reducing the number of parameters by 1.04 Mb. The experimental results demonstrate that weight sharing has a positive impact on feature extraction from multimodal images. The paper infers that since PET and CT scans are spatially aligned, the feature representations of the two scans can complement each other in SMDM. Weight sharing helps reduce false positives and false negatives when delineating tumor regions.

[0132] Table 9 Comparative experimental results of SMDM on the hector 2022 dataset

[0133]

[0134] Specifically, in this embodiment, see Figure 11-12Multimodal tumor segmentation methods can be divided into three types based on their fusion strategies: early fusion methods, late fusion methods, and hybrid fusion methods. Early fusion methods use multimodal image concatenation as model input and use customized modules to extract rich multimodal features. This method typically adopts a fixed fusion ratio and ignores spatially varying visual features, such as the changing importance of different regions. Late fusion methods use multiple branches to extract high-level semantic features from PET / CT images separately, and then integrate these multimodal features through a dedicated fusion module to achieve accurate tumor segmentation. This method may ignore the correlation between the two modalities during feature extraction and lack cross-modal feature interaction. Hybrid fusion methods involve fusing feature maps from other modalities into the model's feature extraction or image reconstruction process. This enables the model to obtain additional feature representations at different stages, ultimately improving segmentation performance. However, this method does not fully exploit the correlation and interaction of multimodal data. The discriminative features of the two modal images should be complementary. In addition, it lacks a global understanding of image features, which may affect the accuracy and generalization of the model. At the same time, existing tumor boundary segmentation methods only focus on extracting boundary features at the model input or output layer, ignoring the potential multi-stage boundary features. This paper proposes a dual-branch tumor automatic segmentation network (DGCBG-Net) based on global cross-modal interaction and boundary guidance for PET / CT image segmentation. In order to extract global context features from PET and CT modalities and promote bilateral cross-modal interaction of global features, this paper proposes a global cross-modal interaction module; in order to help the encoder improve the effectiveness of extracting cross-modal complementary features and reduce feature loss in the downsampling process, this paper proposes a shared multi-path downsampling module. Finally, a boundary prior-guided branch is proposed to extract multi-stage potential boundary features to assist SSB in tumor edge area segmentation; experimental results on the STS dataset and Hecktor 2022 dataset verify the effectiveness of the three basic components proposed in this paper. In addition, compared with existing tumor segmentation methods, the dual-branch tumor automatic segmentation network exhibits superior performance in most segmentation indicators.

[0135] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present description and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A dual-branch tumor automatic segmentation network, characterized in that: include: Semantic segmentation branch and boundary prior guidance branch; The semantic segmentation branch includes a dual-branch encoder and a single-branch decoder; The dual-branch encoder includes a first encoder block, a global cross-modal interaction module, and a shared multi-path downsampling module; The global cross-modal interaction module is used to extract global context features of PET / CT images and perform cross-modal feature interaction; the shared multi-path downsampling module is used to downsample the global context features and extract high-order cross-modal semantic information; the single-branch decoder includes a first decoder block and a transposed convolutional layer; The boundary prior guidance branch includes a second encoder block, a downsampling block, a boundary extraction module and a second decoder block; the boundary extraction module is used to extract boundary features of the CT image.

2. A dual-branch tumor automatic segmentation network according to claim 1, characterized in that: The global cross-modal interaction module includes a CT branch; The CT branch includes a first convolutional layer, a first 2D fast Fourier transform, a first global filter, a first Fourier channel attention block and a first 2D fast inverse Fourier transform; the CT branch is used to extract global context features of the CT image based on the CT image and the boundary features output by the boundary extraction module.

3. A dual-branch tumor automatic segmentation network according to claim 2, characterized in that: The global cross-modal interaction module includes a PET branch; The PET branch includes a second convolutional layer, a second 2D fast Fourier transform, a second global filter, a second Fourier channel attention block and a second 2D fast inverse Fourier transform; the PET branch is used to extract global context features of the PET image based on the PET image.

4. A dual-branch tumor automatic segmentation network according to claim 3, characterized in that: The global cross-modal interaction module extracts global context features of PET / CT images, specifically: The PET image is input into the PET branch, and the CT image and boundary features are input into the CT branch; Perform lossless conversion between frequency domain and spatial domain; Extracting global context features of PET images and CT images, performing element-by-element addition and cross-merging on the global context features, and sharing parameter weights to obtain feature representations of images of different modalities; According to the feature representation of different modal images, the channel features in the frequency domain are obtained based on FCA, and the weight factor of each channel is obtained through the activation function; The feature information fusion of spatial domain and frequency domain is realized through residual connection.

5. A dual-branch tumor automatic segmentation network according to any one of claims 1 to 4, characterized in that: The shared multi-path downsampling module includes a third convolutional layer, a maximum pooling layer, and a channel attention block.

6. A dual-branch tumor automatic segmentation network according to any one of claims 1 to 4, characterized in that: The boundary extraction module includes a Sobel operator, a global filter layer, a convolution layer, a batch processing layer and a ReLU activation.

7. A dual-branch tumor automatic segmentation network according to claim 6, characterized in that: The Sobel operator includes a horizontal boundary detection operator SobelX and a vertical boundary detection operator SobelY; The horizontal boundary detection operator SobelX is used to obtain the horizontal gradient map F of the CT image. bx ; The vertical boundary detection operator SobelY is used to obtain the vertical gradient map F of the CT image. by .

8. A dual-branch tumor automatic segmentation network according to claim 7, characterized in that: The boundary features of the CT image are: Among them, F bxy is the boundary feature of the CT image, Sigmoid() is the activation function, and F bx is the horizontal gradient map of the CT image, F by is the vertical gradient map of the CT image, X ct is the input CT image.

9. A performance evaluation method comprising a dual-branch tumor automatic segmentation network as claimed in any one of claims 1 to 8, characterized in that: The following steps are involved: Set up evaluation datasets and evaluation metrics; Preprocess CT / PET scans and use data augmentation to expand the evaluation dataset; According to the evaluation data set, an ablation experiment is performed on each branch module in the dual-branch tumor automatic segmentation network to determine the effectiveness of each branch module in the dual-branch tumor automatic segmentation network.

10. The performance evaluation method of a dual-branch tumor automatic segmentation network according to claim 9, characterized in that: The evaluation indicators include dice coefficient, intersection-over-union ratio, positive prediction value, recall rate and Hausdorff distance.

Images