An esophageal cancer tumor target segmentation method based on PET / CT image cross-modal feature fusion

Abstract

This invention discloses a method for esophageal cancer tumor target region segmentation based on cross-modal feature fusion of PET / CT images. The method uses a Transformer-fused Attention Progressive Semantic Nested Network (TransAttPSNN) as the 3D segmentation model for esophageal cancer tumor target regions. The TransAttPSNN network has an AttPSNN backbone structure and includes two segmentation networks: one for PET streams and the other for CT streams. Transformer cross-modal adaptive feature fusion modules are embedded in the different feature levels of the two segmentation networks. Compared with existing technologies, this invention effectively improves the segmentation accuracy of esophageal cancer tumor target regions and achieves better segmentation performance.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medical image processing, specifically relating to a method for segmenting esophageal cancer tumor target areas based on cross-modal feature fusion of PET / CT images. Background Technology

[0002] Early-stage esophageal cancer often presents with no obvious symptoms, leading to its diagnosis typically occurring only in advanced stages. For patients with mid-to-late-stage esophageal cancer, radiotherapy is a primary treatment option; it is particularly effective for radiation-sensitive squamous cell carcinoma of the esophagus. The design of a radiotherapy plan relies heavily on the delineation of the esophageal cancer tumor target area. Accurate delineation ensures the tumor receives a sufficient radiation dose during radiotherapy and prevents damage to surrounding normal tissues or dangerous organs from excessive radiation exposure. Currently, the delineation of esophageal cancer tumor target areas is performed manually by physicians. This is a tedious, time-consuming, and labor-intensive task, wasting valuable medical resources. Furthermore, manual delineation relies on the physician's clinical experience and subjective judgment, resulting in variations in the delineated target area for the same patient depending on the physician's assessment, leading to ill-posedness. Therefore, developing effective automatic segmentation algorithms for esophageal cancer tumor target areas using computer-aided technology has become an urgent need.

[0003] In clinical practice, many esophageal cancer patients scheduled for radiotherapy have already undergone PET / CT imaging. Although some techniques currently employ deep learning to segment esophageal cancer tumor target areas, these techniques are not based on PET / CT images, and the accuracy of segmentation still needs improvement. Summary of the Invention

[0004] To fully utilize the complementary information of functional metabolic imaging PET and anatomical structural imaging CT, this invention aims to provide a more accurate and effective method for segmenting esophageal cancer tumor target areas based on PET / CT image processing.

[0005] The specific technical solution of the present invention is as follows.

[0006] A method for esophageal cancer tumor target region segmentation based on cross-modal feature fusion of PET / CT images, specifically including the following steps:

[0007] S1. Collect PET / CT images of clinical esophageal cancer patients and their corresponding labels to form a dataset;

[0008] S2. Preprocess the PET / CT image dataset;

[0009] S3. Establish a three-dimensional segmentation model for esophageal cancer tumor target area: Transformer using Attention Progressive Semantically-Nested Network (TransAttPSNN);

[0010] TransAttPSNN uses the Attention Progressive Semantically-Nested Network (AttPSNN) as its backbone structure. AttPSNN introduces a convolutional attention mechanism into the Progressive Semantically-Nested Network (PSNN). It includes two segmentation networks: one for the PET stream and the other for the CT stream. The network structures of the PET and CT streams are the same. At the same time, Transformer cross-modal adaptive feature fusion modules are embedded in the five feature layers of the two segmentation networks at different scales. Between the PET and CT streams, the five Transformer cross-modal adaptive feature fusion modules are used to connect the PET and CT feature images at five different scales to perform adaptive feature fusion. The fused results are then transmitted back to the PET and CT stream paths to participate in the forward propagation of subsequent information. Through a deep supervised connection method, the outputs of the upper and lower AttPSNN decoding paths and the total outputs of the two paths are first channel-connected, then processed by a convolutional layer, and finally the segmentation prediction result is obtained through the output layer Sigmoid.

[0011] S4. Train the established TransAttPSNN segmentation model;

[0012] S5. Apply the trained TransAttPSNN segmentation model to predict the esophageal cancer tumor target area segmentation in PET / CT images of patients with unknown esophageal cancer, output the best segmentation accuracy, and visualize the segmentation results.

[0013] In this invention, in step S1, the label corresponding to the PET / CT image is determined by importing the PET / CT DICOM file into ITK-SNAP software, referring to the corresponding PET image, and manually delineating and reviewing the esophageal cancer tumor target area on the CT axial slice.

[0014] In this invention, step S2, data preprocessing includes three operations: secondary registration of PET / CT images to correct positional deviations between PET / CT images, contrast enhancement of CT images, and cropping and normalizing regions of interest in PET / CT images. Preferably, the secondary registration of PET / CT images employs a multimodal intensity three-dimensional registration algorithm, a registration method based on mutual information, a registration method based on optical flow field, or a registration method based on deep learning. The PET images output after registration have the same size as the CT images. The CT images are truncated at window width to enhance contrast.

[0015] In this invention, in step S3, the TransAttPSNN network uses the AttPSNN network for both the PET and CT streams. Each AttPSNN network contains feature images of five different scales. Specifically, the encoding path contains five convolutional layers, with the first two layers each consisting of two convolutional modules and the last three layers each consisting of three convolutional modules. The decoding path contains four convolutional attention layers, with the first layer consisting of a convolutional attention module and a ConV layer, and the last three layers consisting of a convolutional attention module, a ConV layer, and a trilinear interpolation upsampling layer.

[0016] In this invention, step S4 employs a four-fold cross-validation method to train the established TransAttPSNN segmentation model. The four-fold cross-validation training method specifically involves: first, dividing the dataset into four equal parts; second, sequentially selecting one part of the data as the test set, and merging the remaining three parts as the training set to train the TransAttPSNN model. Therefore, a total of four TransAttPSNN segmentation models will be trained.

[0017] In summary, this invention addresses the issue of positional deviation in PET / CT images by performing secondary registration. It also addresses the problem of poor contrast in CT images by using statistical analysis to select a reasonable window width truncation threshold for CT image truncation, thereby improving image contrast. Furthermore, this invention proposes a method for esophageal cancer tumor target region segmentation based on dual-modal PET / CT images. This invention innovatively introduces the Transformer model into the 3D segmentation task of esophageal cancer tumor targets, achieving cross-modal adaptive feature fusion, and subsequently proposing a 3D segmentation model for esophageal cancer tumor targets, which integrates the Transformer with the attention-based progressive semantic nested network TransAttPSNN.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] In the TransAttPSNN segmentation model of this invention, the introduction of a convolutional attention mechanism makes the proposed AttPSNN model more effective than the existing Progressive Semantically-Nested Network (PSNN) model. Furthermore, by mining and fusing complementary information from both PET and CT modalities using the Transformer model, the segmentation performance is further improved. Therefore, compared to the state-of-the-art methods reported in the existing literature, the TransAttPSNN 3D segmentation model for esophageal cancer tumors designed in this invention achieves better segmentation accuracy.

[0020] This invention achieves better positional deviation correction through secondary registration of PET / CT images. Selecting a reasonable window width truncation threshold using statistical analysis for CT image truncation improves CT image contrast. This invention innovatively introduces the Transformer model and proposes a method for esophageal cancer tumor target region segmentation based on cross-modal feature fusion of PET / CT images. This method improves the segmentation accuracy of esophageal cancer tumor target regions and can provide technical support for automatic segmentation of esophageal cancer tumor target regions. Attached Figure Description

[0021] Figure 1 This is a comparison of the results before and after secondary registration of PET / CT images according to the present invention; comparison of the results before and after secondary registration of PET / CT images. (a) and (b) are the visualization results of PET / CT image overlay from two different embodiments. The green image represents PET, the purple image represents CT, the blue outline represents the outline of the real label, and the green highlighted area in the blue outline represents the tumor lesion area in PET.

[0022] Figure 2 This is a comparison image showing the results of contrast enhancement of CT images before and after the present invention.

[0023] Figure 3 This is a framework diagram of the TransAttPSNN three-dimensional segmentation model for esophageal cancer tumor target area designed in this invention.

[0024] Figure 4 A diagram illustrating the self-attention weighted correlation matrix for fusing PET and CT feature images.

[0025] Figure 5 The images show the three-dimensional visualization results of the segmentation results obtained by the method of the present invention. (a) is a three-dimensional visualization view of the esophageal cancer tumor corresponding to the real label, (b) is a three-dimensional visualization view of the segmentation results obtained by the present invention, and (c) is an overlay view of (a) and (b). Detailed Implementation

[0026] The implementation details of the present invention will be described in detail below with reference to the accompanying drawings and embodiments, but the scope of protection of the present invention is not limited to the embodiments described.

[0027] A method for esophageal cancer tumor target region segmentation based on cross-modal feature fusion of PET / CT images, specifically including the following steps:

[0028] S1. Collect PET / CT images and their corresponding labels from clinical esophageal cancer patients to form a dataset. The PET / CT images of esophageal cancer patients are DICOM data from whole-body 18F-FDG PET / CT imaging examinations. The labels corresponding to the PET / CT images were first agreed upon by two physicians in routine clinical practice. The PET / CT DICOM files were imported into ITK-SNAP software (Version 3.6, United States), and the esophageal cancer tumor target area was manually delineated on the CT axial slices by referring to the corresponding PET images. Then, one physician reviewed the delineated labels to determine the final labels.

[0029] S2. Preprocess the PET / CT image dataset. The specific preprocessing operations include the following three aspects:

[0030] S2.1 Secondary registration of PET / CT images. Although the PET / CT scanner has performed hardware registration of the images, involuntary respiratory movements, abdominal organ peristalsis, and heartbeats during image acquisition mean that the PET / CT images are not strictly registered. Therefore, we perform secondary registration of PET / CT images in the data preprocessing step to correct positional deviations between PET / CT images. The registration method used is a multimodal intensity three-dimensional registration algorithm. [1] The registered PET and CT images have the same size, 512×512. Note: In addition to the multimodal intensity 3D registration algorithm, methods based on mutual information, optical flow, and deep learning can also be used for PET / CT image registration. [2-4] .like Figure 1 The figure shown is a comparison of the results of secondary registration of PET / CT images in two embodiments of the present invention before and after the secondary registration. It can be observed that the positional deviation of the PET / CT images is well corrected after secondary registration.

[0031] S2.2 performs contrast enhancement on CT images. Specifically, it truncates the CT image window, assigning pixel values ​​less than -150 and greater than 150 in the CT image matrix to -150 and 150 respectively. For example... Figure 2The image shown is a comparison of the results of contrast enhancement of CT images in a certain embodiment of the present invention before and after the enhancement. It can be observed that after window width truncation of the CT images, the contrast between the tumor lesion area and its surrounding tissues in the CT images is improved.

[0032] S2.3 performs region of interest cropping and normalization on PET / CT images. This is limited by the high computational and storage costs of deep neural network models, and also aims to alleviate the problem of significant data imbalance between foreground and background (foreground representing tumor areas, background representing non-tumor areas). [5] Cropping regions of interest (ROIs) from PET / CT images is necessary. Specifically, the PET / CT images and their corresponding labels for each patient in the dataset are first cropped into ROIs containing the esophageal cancer tumor and with a size of at least 64×64×64 pixels. Then, the cropped PET / CT images are normalized to the [0, 1] interval. After cropping and normalizing the ROIs of the PET / CT images, the dataset required for training the network model is obtained.

[0033] S3. Establish a 3D segmentation model for the esophageal cancer tumor target area: This involves using a Transformer-based attention-based progressive semantic nested network and TransAttPSNN. The specific operations include the following two aspects:

[0034] S3.1 uses the Progressive Semantic Nested Network (PSNN) as the backbone in the state-of-the-art three-dimensional segmentation method for esophageal cancer tumor target volume reported in existing literature. [6] They proposed the AttPSNN, an attention-progressive semantic nesting network, by introducing a convolutional attention mechanism.

[0035] S3.2 Design the Transformer cross-modal adaptive feature fusion module and embed it into different scale feature layers of two AttPSNN segmentation networks (one is the PET segmentation stream and the other is the CT segmentation stream) to build the final segmentation model TransAttPSNN.

[0036] like Figure 3The diagram shows the framework of the TransAttPSNN model. Dual-channel PET / CT images are input into the TransAttPSNN network and split into two streams: the upper PET stream and the lower CT stream. The PET and CT streams share the same network structure, using the proposed AttPSNN network. Each AttPSNN network contains feature images at five different scales. Specifically, the encoding path contains five convolutional layers: the first two layers each consist of two convolutional modules (ConV+BN+ReLU), and the last three layers each consist of three convolutional modules; the fifth convolutional layer resembles the middle bridging section of a U-shaped network. The decoding path contains four convolutional attention layers: the first layer consists of a convolutional attention module + a ConV layer, and the last three layers consist of a convolutional attention module + ConV + a trilinear interpolation upsampling layer. Between the PET and CT streams, five Transformer cross-modal adaptive feature fusion modules are used to connect the five PET and CT feature images at different scales for adaptive feature fusion. The fused results are then transmitted back to the PET and CT streams respectively to participate in subsequent forward propagation. Through deep supervised connectivity, the outputs of the upper and lower AttPSNN decoding paths, as well as their respective total outputs, are first channel-connected and then processed by a convolutional layer. Finally, the segmentation prediction result is obtained through the output sigmoid layer.

[0037] In this invention, the function of cross-modal adaptive feature fusion using the Transformer model is implemented as follows.

[0038] 3.2.1 Theory of Three-Dimensional Transformer Model

[0039] Assume the input image is x∈R H′×W′×D′×C Here, H′, W′, D′, and C represent the image's height, width, depth, and number of channels, respectively. To avoid memory explosion, the input image x is first pooled to x using the three-dimensional adaptive average pooling function AdaptiveAvgPool3d(·). pooling ∈R H×W×D×C Where H, W, D, and C are the height, width, depth, and number of channels of the pooled image.

[0040] x pooling =AdaptiveAvgPool3d(x).(1)

[0041] Secondly, use a window of size 1×1×1×C to view x pooling Flattening into a series of patches to obtain x f ∈R (HWD)×CWhere HWD is the number of patches generated, and C is the dimension of each patch. x f When input into a standard Transformer module, the process in the Multi-Head Self-Attention (MHSA) module is as follows:

[0042] q = x f ·W q k = x f ·W k v = x f ·W v (2)

[0043]

[0044]

[0045] z = Concat(z) (1) ;z (2) ;...;z (M) )·W o (5)

[0046] Among them, W q W k W v ∈R C×C Let R represent the mapping matrix. q, k, v ∈ R (HWD)×C Let represent query, key, and value respectively; M is the number of parallel self-attention heads in MHSA. Let d = C / M be the dimension of each self-attention head, then... Let be the mapping matrix for the m-th self-attention head; correspondingly, q m k m v m ∈R (HWD)×d (m = 1, 2, ..., M) represents the query, key, and value of the m-th self-attention head. σ(·) denotes the Softmax function, z (m) ∈R (HWD)×d W is the output of the m-th self-attention head. o ∈R Md×C (i.e. W) o ∈R C×C Let be the mapping matrix, z∈R (HWD)×C This is the final output of MHSA.

[0047] Third, the output of MHSA is sent to the Multi-Layer Perception (MLP) module for processing (MLP consists of two fully connected linear layers, two Dropout layers, and one GELU activation layer).

[0048] Fourth, based on the above process, add a trainable location encoding P. f After the LN layers and residual connection operations, the processing flow for a 3D Transformer model with L Transformer modules is as follows:

[0049] z0 = x f +P f (6)

[0050] z′ l =MHSA(LN(z) l-1 ))+z l-1 (7)

[0051] z l =MLP(LN(z′) l ))+z′,(l=1,2,...,L).(8)

[0052] Finally, z L ∈R (HWD)×C Remodeling In the form of, and then applying trilinear interpolation to... Upsampling This allows you to restore the Transformer's output to the same size as the original input image.

[0053] 3.2.2 Cross-modal adaptive feature fusion theory based on 3D Transformer model

[0054] Based on the above three-dimensional Transformer model theory, assuming the PET and CT feature images to be fused are x PET ∈R H′×W′×D′×C With x CT ∈R H′×W′×D′×C Where H′, W′, D′, and C represent the height, width, depth, and number of channels of the image, respectively. Then, first use equation (1) to process the input image x... PET With x CT Pooling to and Where H, W, D, and C are the height, width, depth, and number of channels of the pooled image.

[0055] Secondly, use a window of size 1×1×1×C to... and Flatten each patch into a series of patches to obtain and Then and The connection is obtained by performing a join along the patch dimension.

[0056] Third, according to equations (6)-(8), Input the Transformer model and process it to obtain the output z L ∈R (2HWD)×C z L Remodeling In the form of, and split into two outputs. and Finally, trilinear interpolation was applied to... and Upsample each image to the same size as the original input image to obtain and This completes the adaptive fusion of PET and CT feature images, and the fusion process is explained in detail below:

[0057] Will After inputting the Transformer model, when processing it in the MHSA module, refer to Equation (4) to calculate the self-attention weights W. a This can be viewed as calculating the correlation between pairs of patches in PET and CT feature images after flattening and mapping, such as... Figure 4 As shown, where w ij (i,j=1,2,...,2HWD) represents the correlation between the patch at position i and the patch at position j. Therefore, the Transformer model can adaptively model the long-range dependencies between PET and CT feature images in the same modality and across modalities during the training process, thereby realizing the feature fusion function.

[0058] S4. The established segmentation model TransAttPSNN is trained using a four-fold cross-validation method. The specific operations include the following four aspects:

[0059] S4.1 divides the dataset into four equal parts.

[0060] S4.2 describes the training configuration for the segmentation model TransAttPSNN. This includes the following two aspects:

[0061] S4.2.1 Randomly extract 16 training patches of size 64×64×64 from each region of interest and its corresponding label obtained in the data preprocessing step S2.3, and randomly perform a data augmentation operation on each patch (rotate 90°, flip left and right, flip up and down, flip left and right first and then rotate 90°, or keep it unchanged).

[0062] S4.2.2 Configure the hyperparameters of the TransAttPSNN segmentation model established in step S3: epochs = 50, learning rate = 5e-3, mini-batch size = 4, optimizer is AdamW, decoupled weight decay = 0.01, and loss function is Generalized Dice Loss (GDL). The formula for GDL is defined as follows:

[0063]

[0064] Where c represents the number of categories, y cn With p cn Let represent the true label value and the predicted probability value of the nth pixel belonging to the cth class, respectively. This represents the reciprocal of the sum of the squares of the total number of pixels in the foreground and background (foreground representing the tumor region, background representing the non-tumor region). Therefore, the weight w c The introduction of ε corrects the relative contributions of the foreground and background, thus alleviating the imbalance problem between foreground and background data. ε = 1 × 10 -8 This is used to prevent the denominator from being zero.

[0065] S4.3 From the four equal parts of the dataset, one part is sequentially taken as the test set, and the remaining three parts are combined as the training set to train the TransAttPSNN model. When training the network, refer to... Figure 3 The data is processed using a small batch (mini-batch=4) of samples, and the specific operations include the following four aspects:

[0066] S4.3.1 Input dual-channel PET / CT images into the established TransAttPSNN model.

[0067] S4.3.2 Extract the PET channel image and send it to the PET stream for processing, and simultaneously extract the CT channel image and send it to the CT stream for processing.

[0068] S4.3.3 For PET and CT feature images of the same level, the Transformer adaptive feature fusion module is used to fuse the two, and the fusion result is fed back to the PET stream and CT stream respectively to participate in information feedforward.

[0069] S4.3.4 By continuously optimizing the training network, a high-performance 3D segmentation model for the esophageal cancer tumor target area is obtained through convergence. It should be noted that due to the four-fold cross-validation training method, a total of four TransAttPSNN segmentation models will be trained.

[0070] S4.4 inputs the four test sets into the TransAttPSNN model trained on their corresponding training sets and outputs the average segmentation accuracy.

[0071] In step S4.4, when testing the network, refer to Figure 3 The data is processed on a single sample basis, and the specific operations include the following four aspects:

[0072] S4.4.1 Input dual-channel PET / CT images into the trained TransAttPSNN model.

[0073] S4.4.2 Extract the PET channel image and send it to the PET stream for processing, and simultaneously extract the CT channel image and send it to the CT stream for processing.

[0074] S4.4.3 For PET and CT feature images of the same level, the Transformer adaptive feature fusion module performs fusion processing on the two and feeds the fusion results back to the PET stream and CT stream respectively to participate in subsequent information feedforward.

[0075] S4.4.4 continuously feeds information forward until the corresponding segmentation result is output.

[0076] In step S4.4, segmentation accuracy is measured using three commonly used evaluation metrics: Dice Similarity Coefficient (DSC), Hausdorff Distance (HD), and Mean Surface Distance (MSD). Hausdorff Distance is also known as the maximum surface distance. DSC measures the degree of spatial overlap between the predicted and ground truth labels. [9,10] The distance metrics HD and MSD measure the maximum distance between the predicted tumor region margin and the actual tumor region margin, respectively, compared with the mean distance.

[11] Assuming the predicted tumor region is represented by P and the actual tumor region by G, their corresponding edge contours are represented by P0 and P1 respectively.C and G C The calculation formulas for DSC, HD, and MSD are defined as follows:

[0077]

[0078]

[0079]

[0080] Where d(p,g) represents the Euclidean distance between pixels p and g; |P| and |G| represent the total number of pixels in the predicted and actual tumor regions P and G, respectively; similarly, |P C |and|G C | represents the total number of pixels that match the predicted tumor edge contour. DSC values ​​range from [0, 1], with values ​​closer to 1 indicating better segmentation results. HD and MSD values ​​are greater than or equal to 0, with values ​​closer to 0 indicating better segmentation results.

[0081] S5. Apply the trained TransAttPSNN segmentation model to predict the segmentation accuracy of PET / CT images of patients with unknown esophageal cancer, and visualize the segmentation results. The specific operations include the following four aspects:

[0082] S5.1 Acquire DICOM data from PET / CT scans of patients with unknown esophageal cancer.

[0083] S5.2 Use the data preprocessing method in step S2 to preprocess the acquired PET / CT images.

[0084] S5.3 inputs the preprocessed PET / CT region of interest image into the four trained TransAttPSNN segmentation models to output the corresponding four sets of segmentation accuracy.

[0085] S5.4 From the four groups of segmentation accuracies obtained in step S5.3, select the group with the largest DSC value as the optimal segmentation accuracies, and visualize the corresponding segmentation results. (Reference) Figure 5 , Figure 5 (a) is a 3D visualization view of the esophageal cancer tumor corresponding to the actual label. Figure 5 (b) is a three-dimensional visualization view of the segmentation results obtained in this invention. Figure 5 (c) is an overlay view of (a) and (b). Observation of the 3D visualization results shows that the esophageal cancer tumor predicted by this invention has a smoother shape than the tumor corresponding to the actual label, which is closer to the actual appearance of the lesion in clinical practice. Furthermore, the esophageal cancer tumor predicted by this invention has a good similarity to the actual tumor.

[0086] Table 1. Comparison of segmentation accuracy between the method of the present invention and other existing esophageal cancer tumor segmentation methods.

[0087]

[0088] Table 1 compares the segmentation accuracy of the TransAttPSNN 3D segmentation model for esophageal cancer tumor target region designed in this invention with other existing esophageal cancer tumor segmentation methods. In Table 1, the benchmark methods based on convolution include U-Net, DenseUNet, and Two-stream chained PSNN; the methods based on convolutional attention include Attention U-Net and DDAUNet; and the methods based on Transformer models include UNETR, TransBTS, and CoTr. Among these, Two-stream chained PSNN, DenseUNet, and DDAUNet represent the most advanced methods in the current literature on esophageal cancer GTV 3D segmentation. By observing and comparing the evaluation index values, it can be seen that the TransAttPSNN network outperforms all other competing networks, achieving the highest DSC value and the lowest HD value. Although its MSD value is slightly lower than that of UNETR, the difference is minimal. Compared to convolutional network methods, Transformer-based methods achieve better segmentation performance (with the exception of CoTr). Among the Transformer-based models, the TransAttPSNN designed in this invention performs best.

[0089] References

[0090] [1]MUTHUKUMARAN D,SIVAKUMAR M.Medical Image Registration: A matlabbased approach[J].Int J Sci Res Comput Sci,Eng Inform Technol,2017,2(2):29-34.

[0091] [2]PENNEC

[0092] [3] Luo Shuqian, Li Xiang. Multimodal medical image registration based on maximum mutual information [J]. Journal of Image and Graphics, 2000, 5(7): 551-8.

[0093] [4]HU Y, MODAT M, GIBSON E, et al. Weakly-supervised convolutional neural networks for multimodal image registration[J]. Med Image Anal, 2018, 49: 1-13.

[0094] [5] CRUM WR, CAMARA O, HILL D L. Generalized overlap measures forevaluation and validation in medical image analysis[J]. IEEE T Med Imaging, 2006, 25(11): 1451-61. [6] JIN D, GUO D, HO TY, et al. DeepTarget: Gross tumor and clinical target volume segmentation in esophageal cancer radiotherapy[J]. MedImage Anal,2021,68:101909.[7]RAJON DA,BOLCH W E.Marching Cube Algorithm:Review and trilinear interpolation adaptation for image-based dosimetricmodels[J].Comput Med Imag Grap,2003,27(5):411-35.

[0095] [8]HILL S.Trilinear Interpolation[J].Graphics Gems,1994:521-5.

[0096] [9]RAZZAK M I,IMRAN M,XU G.Efficient brain tumor segmentation withmultiscale two-pathway-group conventional neural networks[J].IEEE J BiomedHealth,2019,23(5):1911-9.

[0097]

[10] CHEN G,YIN J,DAI Y,et al.A novel convolutional neural network forkidney ultrasound images segmentation[J].Comput Meth Prog Bio,2022,218:106712.

[0098]

[11] FECHTER T,ADEBAHR S,BALTAS D,et al.Esophagus segmentation in CTvia 3D fully convolutional neural network and random walk[J].Med Phys,2017,44(12):6341-52.

[0099]

[12] IEK Z,ABDULKADIR A,LIENKAMP S S,et al.3D U-Net:Learning densevolumetric segmentation from sparse annotation[C].In Proceedings of theInternational Conference on Medical Image Computing and Computer-AssistedIntervention,2016,424-32.

[0100]

[13] FECHTER T,ADEBAHR S,BALTAS D,et al.A 3D fully convolutionalneural network and a random walker to segment the esophagus in CT[J / OL]2017,1-23,arXiv:1704.06544.

[0101]

[14] OKTAY O,SCHLEMPER J,FOLGOC L L,et al.Attention U-Net:Learningwhere to look for the pancreas[C].In Proceedings of the InternationalConference on Medical Imaging with Deep Learning,2018,1-10.

[0102]

[15] YOUSEFI S,SOKOOTI H,ELMAHDY M S,et al.Esophageal gross tumorvolume segmentation using a 3D convolutional neural network[C].In Proceedingsof the International Conference on Medical Image Computing and Computer-Assisted Intervention,2018,343-51.

[0103]

[16] YOUSEFI S,SOKOOTI H,ELMAHDY M S,et al.Esophageal tumorsegmentation in CT images using a dilated dense attention Unet(DDAUnet)[J].IEEE Access,2021,9:99235-48.

[0104]

[17] HATAMIZADEH A,TANG Y,NATH V,et al.UNETR transformers for 3Dmedical image segmentation[C].In Proceedings of IEEE Winter Conference onApplications of Computer Vision,2021,1-11.

[0105]

[18] WANG W,CHEN C,DING M,et al.TransBTS:Multimodal brain tumorsegmentation using transformer[C].In Proceedings of the InternationalConference on Medical Image Computing and Computer-Assisted Intervention,2021,1-11.

[0106]

[19] XIE Y,ZHANG J,SHEN C,et al.CoTr:Efficiently bridging CNN andtransformer for 3D medical image segmentation[J / OL]2021,1-13,arXiv:2103.03024。

Claims

1. A method for esophageal cancer tumor target region segmentation based on cross-modal feature fusion of PET / CT images, characterized in that, Includes the following steps: S1. Collect PET / CT images of clinical esophageal cancer patients and their corresponding labels to form a PET / CT image dataset; S2. Preprocess the PET / CT image dataset; S3. Establish a three-dimensional segmentation model for esophageal cancer tumor target area: Transformer fused with attention-based progressive semantic nesting network TransAttPSNN; The TransAttPSNN network uses the AttPSNN progressive semantic nesting network, which incorporates a convolutional attention mechanism, as its backbone. It includes two segmentation networks: one for the PET stream and the other for the CT stream. The PET and CT streams share the same network structure, both employing the AttPSNN network. The AttPSNN network has a U-shaped structure, built by introducing a convolutional attention mechanism into the progressive semantic nesting network PSNN. It contains feature images at five different scales, and the encoding path contains five convolutional layers. The first two convolutional layers each consist of two cascaded convolutional modules, and the last three convolutional layers each consist of three cascaded convolutional modules. Each convolutional module consists of a ConV layer, a Batch Normalization (BN) layer, and a Rectified Linear Activation (ReLU) layer. Max pooling layers are connected between the five convolutional layers in the encoding path for downsampling. The fifth convolutional layer... This is the middle bridging part of the U-shaped network structure; the decoding path contains 4 convolutional attention layers, where the first convolutional attention layer consists of a convolutional attention module connected in series with a ConV layer, and the latter three convolutional attention layers consist of a convolutional attention module connected in series with a ConV layer and a trilinear interpolation upsampling layer; between the PET stream and the CT stream, 5 Transformer cross-modal adaptive feature fusion modules are used to connect 5 PET and CT feature images of different scales to perform adaptive feature fusion. The fused results are then transmitted back to the PET stream and CT stream paths to participate in the subsequent forward propagation of information. Through a deep supervised connection method, the outputs of the upper and lower AttPSNN decoding paths and the total outputs of the two paths are first channel-connected, then processed by a convolutional layer, and finally the segmentation prediction result is obtained through the output layer Sigmoid; S4. Train the established TransAttPSNN segmentation model; S5. The trained TransAttPSNN segmentation model is applied to PET / CT images of patients with unknown esophageal cancer to predict the segmentation of the esophageal cancer tumor target area, output the best segmentation accuracy, and visualize the segmentation results.

2. The method for segmenting the esophageal cancer tumor target area according to claim 1, characterized in that, In step S1, the labels corresponding to the PET / CT images are determined by importing the PET / CT DICOM file into the ITK-SNAP software, referring to the corresponding PET images, and manually delineating and reviewing the esophageal cancer tumor target area on the CT axial slices.

3. The method for segmenting the esophageal cancer tumor target area according to claim 1, characterized in that, In step S2, data preprocessing includes three operations: secondary registration of PET / CT images to correct positional deviations between PET / CT images, contrast enhancement of CT images, and cropping and normalizing regions of interest in PET / CT images.

4. The method for segmenting the esophageal cancer tumor target area according to claim 3, characterized in that, Secondary registration of PET / CT images is performed using a multimodal intensity three-dimensional registration algorithm, a registration method based on mutual information, a registration method based on optical flow field, or a registration method based on deep learning. The PET image output after registration has the same size as the CT image. The CT image is truncated by window width to enhance the contrast of the CT image.

5. The method for segmenting the esophageal cancer tumor target area according to claim 1, characterized in that, In step S4, the TransAttPSNN segmentation model is trained using a four-fold cross-validation method.

6. The method for segmenting the esophageal cancer tumor target area according to claim 1, characterized in that, In step S4, the model training method is as follows: (1) Input dual-channel PET / CT images into the established TransAttPSNN segmentation model; (2) Extract PET channel images and send them to the PET stream for processing, and at the same time extract CT channel images and send them to the CT stream for processing; (3) For PET and CT feature images of the same level, the Transformer adaptive feature fusion module is used to fuse the two, and the fusion result is fed back to the PET stream and CT stream respectively to participate in information feedforward; (4) By continuously optimizing the training network, a high-performance three-dimensional segmentation model of the esophageal cancer tumor target area is obtained through convergence.

7. The method for segmenting the esophageal cancer tumor target area according to claim 1 or 6, characterized in that, In step S4, during training, the optimizer is AdamW, and the loss function is the generalized Dice loss; the formula for GDL is defined as follows: , in, Indicates the number of categories. and They represent the first The pixel belongs to the first The true label value and the predicted probability value of the class This represents the reciprocal of the sum of the squares of the total number of pixels in the foreground and background, where the foreground represents the tumor region and the background represents the non-tumor region. Indicates weight, This is used to prevent the denominator from being zero.

Images