Multi-mode head and neck tumor segmentation and survival prognosis prediction method

Abstract

The invention discloses a multi-mode head and neck tumor segmentation and survival prognosis prediction method. The method comprises the following steps: constructing a DE-LS-UNet model; training the DE-LS-UNet model to obtain a trained DE-LS-UNet model, and inputting the new PET image and the CT image into the trained DE-LS-UNet model to obtain a predicted tumor segmentation mask; extracting multi-modal radiomics features according to the predicted tumor segmentation mask, and fusing clinical features through the multi-modal radiomics features to generate multi-source features of the patient; inputting the multi-source features of the patient into the survival analysis model for training to obtain a trained survival analysis model, and inputting the features of the patient into the trained survival analysis model to obtain a prognosis prediction result. According to the method, a feature extraction and fusion mechanism is provided, and a survival analysis model strategy is combined, so that the stability and prediction performance of survival prognosis modeling can be enhanced while the image segmentation precision is improved.

Description

Technical Field

[0001] This invention belongs to the field of medical assistance and image processing technology, specifically relating to a multimodal head and neck tumor segmentation and survival prognosis prediction method. Background Technology

[0002] Head and neck tumors are common and highly heterogeneous in clinical practice, making early and accurate diagnosis and prognostic assessment crucial for developing clinical interventions. Medical imaging technologies, particularly computed tomography (CT) and positron emission tomography (PET), have become key tools for the imaging diagnosis of head and neck tumors. CT can present high-resolution anatomical structures, while PET can reflect the metabolic activity and functional status of tumor tissue. Therefore, integrating and analyzing multimodal CT and PET images helps to more accurately and comprehensively identify and assess tumors.

[0003] In image analysis methods, 3D U-Net (3D convolutional neural network) based on deep learning has made significant progress in medical image segmentation, and its encoder-decoder architecture can effectively capture the spatial features of tumor targets. However, traditional 3D U-Net still faces technical challenges in handling multimodal data fusion and complex spatial structure modeling: First, current multimodal feature fusion is usually limited to channel stitching or simple convolution stacking, failing to fully explore the complementary information features between CT and PET images; second, the receptive field of standard convolution operations is limited, making it difficult to effectively capture complex spatial relationships of tumors across scales and levels, especially when lesion boundaries are unclear or morphology is complex. In addition, skip connections in U-Net usually directly stitch encoder features with decoder features, lacking a screening mechanism for the importance of low-level features, leading to information redundancy, which in turn affects the segmentation accuracy in the decoding stage.

[0004] In survival prognosis research, the extraction and modeling of radiomics features have been widely used for survival time prediction. However, existing survival analysis methods are mostly separated from segmentation model training, relying solely on manually delineated traditional image segmentation regions for feature extraction, failing to fully utilize the high-order features learned by deep segmentation models and the accurate tumor segmentation results. Therefore, developing a new integrated modeling system that can combine multimodal image automatic segmentation with survival prognosis prediction has become a key challenge and cutting-edge direction in the field of clinical AI-assisted diagnosis.

[0005] Current multimodal image segmentation and survival prognosis prediction technologies for head and neck tumors have significant shortcomings in the following aspects: (1) Insufficient multimodal image fusion capability. Traditional methods often adopt a simple strategy of direct channel stitching for fusion of CT and PET images, which fails to effectively distinguish and enhance the complementary features between modalities, resulting in frequent feature redundancy and information cross-inhibition, affecting the fusion expression capability. (2) Limited spatial context receptive field. Traditional convolutional kernels can only cover spatial information within a fixed scale range, lacking the global perception capability for complex morphological changes and cross-level spatial dependencies of tumors, especially prone to missed detection or false detection in the extraction of irregular tumor boundaries. (3) Coarse utilization of skip connection features. Low-level features in skip connections are directly stitched into the high-level decoding stage without screening, which easily introduces background noise or irrelevant information, weakening the tumor feature focusing ability of the decoding network. (4) The segmentation and survival prognosis analysis processes are independent of each other, lacking a systematic integrated modeling process, failing to integrate the deep spatial information of the segmentation results into the survival analysis model, limiting the accurate prediction and generalization capabilities of the survival prognosis model. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides a multimodal head and neck tumor segmentation and survival prognosis prediction method that solves the problems mentioned above in existing head and neck tumor multimodal image segmentation and survival prognosis prediction technologies.

[0007] To achieve the aforementioned objectives, the present invention employs the following technical solution: a multimodal head and neck tumor segmentation and survival prognosis prediction method, comprising the following steps: S1. Construct the DE-LS-UNet model. The specific process is as follows: Based on the 3D U-Net model, a neighborhood-aware attention module is set in the encoder part of the fourth and fifth layers. The input of each neighborhood-aware attention module is connected to the output of the 3D max pooling operation of the previous layer, and the output of each neighborhood-aware attention module is connected to the input of the convolution module of the encoder part of that layer. A jump attention gating module is set after the jump connection path of the decoder part of the first to fourth layers. The input of each jump attention gating module is connected to the output of the jump connection path and the output of the convolution module of the decoder part of the next layer, respectively. The output of each jump attention gating module is connected to the input of the convolution module of the decoder part of that layer. S2. Input the registered PET and CT images into the DE-LS-UNet model, train the DE-LS-UNet model to obtain a trained DE-LS-UNet model, input the new PET and CT images into the trained DE-LS-UNet model to obtain the predicted tumor segmentation mask. S3. Extract multimodal radiomics features based on the predicted tumor segmentation mask, and fuse clinical features with multimodal radiomics features to generate multi-source patient features; S4. Input the patient's multi-source features into the survival analysis model, train the survival analysis model to obtain a trained survival analysis model, input the patient's features into the trained survival analysis model, and obtain the prognostic prediction results.

[0008] Furthermore: In S1, the specific workflow of the DE-LS-UNet model is as follows: S11. Input the CT image and PET image into the DE-LS-UNet model. Perform convolution operations on both the CT image and the PET image to generate intermediate feature maps. Specifically, the convolution operation involves performing two consecutive convolution module processes. The convolution module processes are performed sequentially by performing a 3×3×3 three-dimensional convolution operation, batch normalization, and PReLU activation function processing. Perform three-dimensional max pooling operation on the intermediate feature map to obtain the CT feature map and the PET feature map. S12. Perform convolution operations on both the CT feature map and the PET feature map to generate intermediate feature maps. Then, stitch and fuse the two intermediate feature maps in the channel dimension. Perform three-dimensional max pooling operation on the fusion result to obtain the first feature map. S13. Perform two consecutive convolution operations, a three-dimensional max pooling operation, and a neighborhood-aware attention module on the first feature map to obtain the second feature map, and generate an intermediate feature map with each convolution operation. S14. Perform a convolution operation on the second feature map to obtain the third feature map. Perform four consecutive three-dimensional upsampling, skip attention gating module processing and convolution operations on the third feature map to obtain the fourth feature map. After each three-dimensional upsampling, input the intermediate feature map of this layer into the skip attention gating module through the skip connection path. S15. Perform a 1×1×1 three-dimensional convolution operation on the fourth feature map to obtain the output of the DE-LS-UNet model.

[0009] The beneficial effects of the above-mentioned further solutions are as follows: This invention proposes an innovative segmentation framework—DE-LS-UNet. This network introduces structural improvements based on the standard U-Net: it employs independent dual-branch encoders to extract features from CT and PET images separately, and fuses two key attention mechanism modules to enhance tumor region recognition capabilities. By fully utilizing the complementary characteristics of CT and PET images in terms of anatomical structure and metabolic function, an independent dual-branch encoder structure is constructed to extract features from the two modalities separately before deep fusion, effectively improving the model's ability to discriminate complex tumor tissues and its robustness.

[0010] Furthermore: In S1, the specific workflow of the neighborhood awareness and attention module is as follows: A1. Perform a 3D convolution operation on the input features of the neighborhood awareness attention module to generate query features and value features. Perform a 3D neighborhood unfolding operation on the value features to extract a subset of neighborhood features. A2. Calculate the similarity score by multiplying each neighborhood feature subset with the query feature at the corresponding position element by element. Normalize all the similarity scores in the domain dimension to obtain the domain weight distribution. A3. Calculate the initial output features based on the domain weight distribution and the neighborhood feature subset. Project the initial output features using a 1×1×1 three-dimensional convolution. Add the projection result to the input feature residuals that have been cropped to the same size as the projection result to obtain the output features of the neighborhood perception attention module.

[0011] The beneficial effects of the above-mentioned further scheme are as follows: a neighborhood-aware attention module is deeply embedded in the U-Net encoder. This module guides the weighted aggregation of local features based on neighborhood information, which enhances the model's ability to express fine-grained structural information of tumors and overcomes the problem of insufficient spatial dependency capture in traditional convolutional models.

[0012] Furthermore: In S1, the specific workflow of the jump attention gating module is as follows: B1. Based on the high-level semantic feature map obtained by 3D upsampling and the intermediate feature map output by the skip connection path, the high-level semantic feature map and the intermediate feature map are reduced to dimensionality by 1×1×1 3D convolution to map features with the same number of intermediate channels. The two map features are added point by point and then activated by the ReLU function to obtain the activated feature map. B2. After the activated feature map is convolved with a 1×1×1 three-dimensional convolution, it is activated by the Sigmoid function to obtain the spatial attention weights shared between channels. The intermediate feature map is multiplied element-wise by the spatial attention weights. The result of the element-wise multiplication is concatenated with the high-level semantic feature map to obtain the output features of the jump attention gating module.

[0013] The beneficial effects of the above-mentioned further scheme are as follows: a skip attention gating module is introduced at the skip connection of each layer of decoder. This module effectively suppresses the interference of redundant and irrelevant background features in the skip connection through a feature selective guidance mechanism, thereby improving the fusion quality of high-resolution semantic information.

[0014] Furthermore, in S2, a combined loss function is used for optimization during the training of the DE-LS-UNet model to simultaneously balance the class imbalance problem and the fineness of segmentation contour boundaries. The specific expression is: In the formula, The Dice loss measures the degree of overlap between the predicted mask and the ground truth label. Let cross-entropy be the loss function. For weight hyperparameters, ; In the formula, For the model in the first i The probability value predicted at the individual element. For the first i The true label of individual factors To prevent smoothing factors with a denominator of zero; In the formula, voxels i Category C The predicted probability, This is a real label.

[0015] Furthermore, S3 includes the following sub-steps: S31. Based on the predicted tumor segmentation mask, extract radiomics features from the corresponding CT and PET images, including intensity statistical features, shape geometric features, texture features and high-level features; S32. Preprocess the radiomics features extracted from CT and PET images to generate CT feature vectors and PET feature vectors. Then, concatenate the CT and PET feature vectors of the same patient along the feature dimension to generate multimodal radiomics features. S33. Process the patient's clinical information to generate clinical features, and then integrate the clinical features with multimodal radiomics features in terms of feature dimensions to generate unified multi-source patient features.

[0016] The beneficial effects of the above-mentioned further scheme are as follows: Based on the segmentation results, combined with radiomics feature extraction and clinical feature processing, a unified high-dimensional multimodal feature matrix is ​​constructed, which integrates the patient's morphological, metabolic, physiological and treatment information, providing comprehensive support for subsequent survival analysis modeling.

[0017] Furthermore, in S4, the survival analysis model consists of an input layer, a hidden layer, and an output layer connected in sequence. The input layer inputs multi-source feature representations; the hidden layer consists of several layers, each with the same structure, including a ReLU activation function layer, a Dropout layer, and a linear layer connected in sequence; the output layer outputs a real-valued risk score.

[0018] Furthermore, in S4, a negative logarithmic loss function is used for optimization during the training of the survival analysis model to effectively handle censored data. The specific expression of the negative logarithmic loss function is as follows: In the formula, For event indicator, 1 indicates a patient relapse, 0 indicates censorship. For follow-up time, In order to be in The sample index set that is still in the risk concentration at all times. Let the risk score of the j-th sample be represented as... exponential form, A single scalar risk score output by the output layer; In the formula, For each patient, multi-source feature expression, It is the ReLU activation function. For Dropout layer, It is a linear layer.

[0019] The beneficial effects of the above-mentioned further solutions are as follows: This invention inputs multi-source patient features into the DeepSurv survival analysis model, optimizes it by minimizing the Cox partial likelihood loss function, effectively models survival time under censored data, predicts relapse-free survival time of patients, and achieves individualized risk scoring.

[0020] The beneficial effects of this invention are as follows: This invention provides a multimodal head and neck tumor segmentation and survival prognosis prediction method, which has the following advantages: (1) Enhanced multimodal feature extraction capability: This invention introduces a dual-branch independent coding structure to achieve parallel extraction and processing of CT and PET images, fully exploring their complementarity at the structural and functional levels. Compared with the traditional single-branch model, this method effectively avoids information interference between modalities and improves the integrity of feature expression and segmentation performance.

[0021] (2) Enhanced local spatial context awareness: By using the proposed neighborhood awareness attention module, the dynamic modeling capability of local neighborhood spatial structure is introduced in the deep stage of the encoder, which effectively captures complex local features such as tumor region boundaries and textures, and improves the model's ability to express tumor heterogeneity, especially when dealing with tumors with complex shapes and blurred boundaries.

[0022] (3) Salience guidance mechanism in skip connection path: In the decoding stage, a skip attention gating module is introduced into each skip connection path to automatically suppress redundant features of non-target areas, strengthen attention to the core tumor area, make the model more discriminative when restoring spatial details, and help improve the accuracy and edge continuity of the overall segmentation results.

[0023] (4) Improved performance across multiple metrics: The proposed DE-LS-UNet model demonstrates excellent performance in the automatic segmentation of head and neck tumors, especially under multimodal fusion conditions. Compared with traditional 3D U-Net and ordinary dual-branch networks, the improved model shows significant improvements in multiple key segmentation metrics. In specific experiments, the model improved the Dice coefficient by approximately 1–2 percentage points, Precision by 2–3 percentage points, and Recall by 1–3 percentage points, while the Hausdorff distance decreased, reflecting its significant enhancement in tumor boundary capture and structural integrity modeling.

[0024] (5) Support for accurate prognostic modeling: This method not only has the ability to accurately segment tumor regions, but also can extract radiomics features by combining the prediction mask with the original image, integrate the patient's clinical parameters, and further use them for modeling and predicting recurrence-free survival time, thus constructing a closed loop from segmentation to survival analysis and improving the integrity and practicality of intelligent tumor diagnosis and treatment. Attached Figure Description

[0025] Figure 1 This is a flowchart of a multimodal head and neck tumor segmentation and survival prognosis prediction method according to the present invention.

[0026] Figure 2 This is a structural diagram of the DE-LS-UNet model of the present invention.

[0027] Figure 3 This is a flowchart of the DE-LS-UNet model of the present invention.

[0028] Figure 4 This is a structural diagram of the neighborhood perception attention module of the present invention.

[0029] Figure 5 This is a structural diagram of the jump attention gating module of the present invention.

[0030] Figure 6 This is a structural diagram of the survival analysis model of the present invention.

[0031] Figure 7 This is a graph showing the changes in the consistency index and Brill score data of this invention.

[0032] Figure 8 This is a typical Kaplan-Meier survival curve diagram of the present invention.

[0033] Figure 9 This is a scatter plot showing the relationship between the survival time and the predicted risk score of this invention. Detailed Implementation

[0034] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0035] This invention aims to address the aforementioned problems in existing multimodal image segmentation and survival prognosis prediction technologies for head and neck tumors by providing an integrated method for automatic multimodal image segmentation and survival prognosis prediction based on an improved 3D U-Net architecture. This invention improves image segmentation accuracy while enhancing the stability and predictive performance of survival prognosis modeling.

[0036] Specifically, the technical solution of this invention proposes a dual-branch independent encoder architecture for segmentation model structure. This architecture independently extracts features from CT and PET modalities initially, and then fuses these features in the later stages of encoding to fully leverage the complementary advantages of multimodal images. A neighborhood-aware attention module is introduced into the multi-level encoder to dynamically adjust the spatial receptive field, effectively modeling long-distance spatial relationships in complex tumor structures and improving the model's ability to capture irregular lesion boundaries and heterogeneous tissues. A skip attention gating module is introduced into the skip connections of the decoder to dynamically regulate the contribution of low-level features, significantly improving the model's ability to finely segment the tumor core region and boundary transition areas.

[0037] Based on the segmentation output, radiomics features are automatically extracted using the predicted tumor mask. Combined with the patient's clinical information features, and after screening and standardization, a survival analysis model based on a deep neural network (DeepSurv) is constructed. This invention achieves the organic integration of tumor segmentation and survival prognosis prediction through an end-to-end integrated architecture, enhancing the intelligent support capability for individualized risk assessment and treatment decision-making in clinical applications.

[0038] like Figure 1 As shown, in one embodiment of the present invention, a multimodal head and neck tumor segmentation and survival prognosis prediction method includes the following steps: S1. Construct the DE-LS-UNet model. The specific process is as follows: Based on the 3D U-Net model, a neighborhood-aware attention module is set in the encoder part of the fourth and fifth layers. The input of each neighborhood-aware attention module is connected to the output of the 3D max pooling operation of the previous layer, and the output of each neighborhood-aware attention module is connected to the input of the convolution module of the encoder part of that layer. A jump attention gating module is set after the jump connection path of the decoder part of the first to fourth layers. The input of each jump attention gating module is connected to the output of the jump connection path and the output of the convolution module of the decoder part of the next layer, respectively. The output of each jump attention gating module is connected to the input of the convolution module of the decoder part of that layer. S2. Input the registered PET and CT images into the DE-LS-UNet model, train the DE-LS-UNet model to obtain a trained DE-LS-UNet model, input the new PET and CT images into the trained DE-LS-UNet model to obtain the predicted tumor segmentation mask. S3. Extract multimodal radiomics features based on the predicted tumor segmentation mask, and fuse clinical features with multimodal radiomics features to generate multi-source patient features; S4. Input the patient's multi-source features into the survival analysis model, train the survival analysis model to obtain a trained survival analysis model, input the patient's features into the trained survival analysis model, and obtain the prognostic prediction results.

[0039] like Figure 2 As shown in this embodiment, the present invention proposes an improved 3DU-Net neural network architecture that integrates multimodal image features: the DE-LS-UNet model. Addressing the feature differences between CT and PET images, it employs a dual-branch independent encoder strategy and integrates a Locality-Aware Attention (LAA) module and a Skip Attention Gate (SAG) module to improve the model's segmentation performance and generalization capabilities for head and neck tumor tissues. This model achieves full extraction and multi-level fusion of feature information through modular design of different attention module insertion positions. The core principle of constructing the DE-LS-UNet model in this invention is as follows: In the encoder, a neighborhood-aware attention module is introduced at the fourth and fifth layers, respectively. This module is placed before each convolutional module and after the max pooling operation, enabling the local spatial awareness mechanism to weighted focus on key spatial regions while maintaining the feature map resolution. The fourth layer, as a mid-to-high-level feature abstraction location, enhances the perception of boundary regions and modal differences through its local attention. The fifth layer, as the deepest layer of the network, further improves the model's ability to express high-level semantics and spatial structures through the introduction of the neighborhood-aware attention module, effectively enhancing the depth of feature extraction.

[0040] Subsequently, skip attention gating modules are introduced into the skip connection paths at each stage of the decoder. Each skip attention gating module receives intermediate feature maps from the encoder and corresponding scale feature maps upsampled by the decoder as inputs. It calculates attention weights through a spatial gating mechanism and uses these weights to modulate the encoder output features before fusing them with the decoder path features. This mechanism can selectively retain and enhance discriminative features from the encoder based on the current semantic context, thereby mitigating the information noise problem caused by feature redundancy or modal inconsistency during multi-scale stitching.

[0041] Based on the above improvements, this embodiment integrates the two modules into the deep layers of the encoder and the skip connections of the decoder, respectively, to construct a multimodal segmentation framework with spatial local perception and cross-layer feature guidance capabilities. While maintaining the advantages of the network's end-to-end symmetrical structure, this structure significantly enhances the network's structural modeling capabilities and target region focusing capabilities through module integration, effectively ensuring the accuracy of tumor region segmentation.

[0042] like Figure 3 As shown, the workflow of the DE-LS-UNet model in S1 is as follows: S11. Input the CT image and PET image into the DE-LS-UNet model. In the encoder part of the first layer, convolution operation is performed on both the CT image and the PET image to generate intermediate feature maps. Specifically, the convolution operation is to perform two consecutive convolution module processes. The convolution module processes are to perform 3×3×3 three-dimensional convolution operation, batch normalization and PReLU activation function processing in sequence. Three-dimensional max pooling operation is performed on the intermediate feature map to obtain the CT feature map and PET feature map. In this embodiment, the DE-LS-UNet model designs two independent encoder paths for the input CT and PET images, respectively. Each path contains several convolutional modules, each consisting of a 3D convolutional unit (Conv3D), batch normalization (BatchNorm3D), a PReLU activation function, and Dropout. These modules sequentially extract the spatial texture, tissue morphology, and image intensity features of each CT and PET image at different scales. By designing independent encoders for CT and PET images, the model effectively captures the differences in image feature representations of lesions across different modalities, avoiding mutual interference between features in the early encoding stages. This preserves the most representative low- and mid-level feature information of both CT and PET images.

[0043] In the first two layers of the encoder, the branches for CT and PET images operate independently, each extracting low-level semantic and edge structure features within its respective modality. Maintaining the independence of feature paths in this stage helps preserve the original expressive power of each modality. Simultaneously, the output features of the encoders in the first two layers are fed into the corresponding decoder layers via skip connections, enhancing the segmentation model's ability to recover fine-grained spatial information. From the third layer onwards, the mid-to-high-level abstract features extracted by the encoders of the CT and PET images are concatenated and fused along the channel dimension, serving as a joint representation input to the subsequent neighborhood-aware attention module and decoder stages, further improving the expressive power of multimodal features and segmentation performance.

[0044] S12. Perform convolution operations on both the CT feature map and the PET feature map to generate intermediate feature maps. Then, stitch and fuse the two intermediate feature maps in the channel dimension. Perform three-dimensional max pooling operation on the fusion result to obtain the first feature map. S13. Perform two consecutive convolution operations, a three-dimensional max pooling operation, and a neighborhood-aware attention module on the first feature map to obtain the second feature map, and generate an intermediate feature map with each convolution operation. S14. Perform a convolution operation on the second feature map to obtain the third feature map. Perform four consecutive three-dimensional upsampling, skip attention gating module processing and convolution operations on the third feature map to obtain the fourth feature map. After each three-dimensional upsampling, input the intermediate feature map of this layer into the skip attention gating module through the skip connection path. S15. Perform a 1×1×1 three-dimensional convolution operation on the fourth feature map to obtain the output of the DE-LS-UNet model.

[0045] In this embodiment, to effectively enhance the model's ability to characterize the complex spatial structure of the tumor region, the present invention introduces a neighborhood-aware attention module into the improved network. The structure of the neighborhood-aware attention module is as follows: Figure 4 As shown, this module draws on and extends the idea of ​​Outlook Attention, and makes adaptive improvements for the characteristics of 3D medical images. Its core idea is to weight and aggregate the information of other voxels in the local area of ​​each voxel at each voxel location. This can fully integrate local neighborhood information during feature extraction, thereby improving the model's ability to characterize tumor boundaries and complex structures and achieving more refined spatial information integration.

[0046] In S1, the workflow of the neighborhood awareness and attention module is as follows: A1. Perform a 3D convolution operation on the input features of the neighborhood awareness attention module to generate query features and value features. Perform a 3D neighborhood unfolding operation on the value features to extract a subset of neighborhood features. like Figure 4 As shown, in this embodiment, for input features ,in B Indicates the batch size. C Indicates the number of channels. D , H and W These represent the depth, height, and width dimensions, respectively. First, the input features are mapped to generate query features through two sets of 1×1×1 convolution operations. Q and value characteristics V , , ; In the formula, This represents a 1×1×1 three-dimensional convolution. Two three-dimensional convolutions map the input features to the query space and the value space, respectively, while keeping the spatial dimensions unchanged.

[0047] logarithmic features V Perform a 3D spatial unfold operation. For each spatial location, extract a subset of neighborhood features centered at that location with a window size of K×K×K. ; In the formula, , , , This is the unfolded size. This indicates a domain expansion operation; the window size K is a hyperparameter, which is usually set to K=3 in this embodiment, that is, local domain information with a radius of 1 is aggregated in each direction.

[0048] A2. Calculate the similarity score by multiplying each neighborhood feature subset with the query feature at the corresponding position element by element. Normalize all the similarity scores in the domain dimension to obtain the domain weight distribution. In this embodiment, for each feature subset within a neighborhood, the corresponding query feature is... Calculate the similarity score by multiplying elements one by one. S , ; In the formula, ⊙ represents element-wise multiplication, and this dot product operation is performed along the channel dimension. ,Will Q Align and crop to the same spatial dimensions to obtain All similarity scores obtained in the domain dimension The neighborhood weight distribution is obtained by normalization using the Softmax function. ; In the formula, , For the Softmax function; A3. Calculate the initial output features based on the domain weight distribution and the neighborhood feature subset. Project the initial output features using a 1×1×1 three-dimensional convolution. Add the projection result to the input feature residuals that have been cropped to the same size as the projection result to obtain the output features of the neighborhood perception attention module.

[0049] In this embodiment, weighted aggregation is used to obtain the output features at each center position, thus obtaining the initial output features. Y ; In the formula, ,right Y Projection is performed using a 1×1×1 3D convolution to obtain the projection result. ; Will Compared with the cropped original input The residuals are summed to form residual connections, which yield the output features of the neighborhood-aware attention module. Z ; In the formula, It is by... X The output features of the neighborhood-aware attention module are obtained by aligning and cropping to the same spatial size. That is, when K When the value is 3, the overall space size shrinks. K-1, keeping the number of channels unchanged.

[0050] like Figure 5 As shown, to improve the quality of skip connection information transmission in multimodal medical images using the U-Net network, this invention introduces a three-dimensional Skip Attention Gate 3D (SAG) module. This module is designed to dynamically suppress irrelevant background features in the skip connections of the U-Net while emphasizing spatial region information related to the target region (GTVp tumor), thereby improving the accuracy of segmentation results and the generalization ability of the model.

[0051] This skip attention gating module introduces a gating mechanism to fuse and judge the low-level feature map from the encoder and the high-level semantic feature map from the decoder, and generates a set of weight masks based on the fusion result to perform weighted suppression or enhancement on the encoder feature map.

[0052] In S1, the specific workflow of the jump attention gating module is as follows: B1. Based on the high-level semantic feature map obtained by 3D upsampling and the intermediate feature map output by the skip connection path, the high-level semantic feature map and the intermediate feature map are reduced to dimensionality by 1×1×1 3D convolution to map features with the same number of intermediate channels. The two map features are added point by point and then activated by the ReLU function to obtain the activated feature map. like Figure 5 As shown, the jump attention gating module has two inputs, from the intermediate feature map of the encoder. x and high-level semantic feature maps from the decoder g , , ,in, and These represent the number of channels, and x and g Dimensionality reduction to the same number of intermediate channels using 1×1×1 3D convolutions. To obtain the mapping features and : In the formula, , , and This corresponds to a 1×1×1 3D convolution; the two mapped features are added point by point and then activated using the ReLU function to obtain the activated feature map. ; In the formula, This is the ReLU function.

[0053] B2. After the activated feature map is convolved with a 1×1×1 three-dimensional convolution, it is activated by the Sigmoid function to obtain the spatial attention weights shared between channels. The intermediate feature map is multiplied element-wise by the spatial attention weights. The result of the element-wise multiplication is concatenated with the high-level semantic feature map to obtain the output features of the jump attention gating module.

[0054] In this embodiment, the activated feature map f The spatial attention weights shared between channels are obtained by performing a 1×1×1 three-dimensional convolution followed by sigmoid activation. , ; In the formula, It is a 1×1×1 three-dimensional convolution. Use the Sigmoid activation function; The weight is used to perform element-wise multiplication on the intermediate feature map to enhance the salient region and suppress the background region. In the formula, The result is the result of element-wise multiplication. The feature dimensions remain consistent with the input encoder features; The result of element-wise multiplication is concatenated with the high-level semantic feature map and used as the output feature of the jump attention gating module. ; In the formula, Indicates channel splicing. .

[0055] In summary, the skip attention gating module of this invention is integrated into the skip connections of the U-Net structure without introducing changes in spatial resolution, while effectively enhancing the model's responsiveness to the target region and improving the expressiveness of segmentation boundaries and detailed structures. This structure, while maintaining the advantages of end-to-end symmetric network structure, significantly improves the network's structural modeling ability and target region focusing ability through module integration, providing effective assurance for the accuracy of tumor region segmentation.

[0056] In this embodiment, after the DE-LS-UNet model structure is completed, the network is further trained and optimized end-to-end to improve its performance in head and neck tumor segmentation tasks. In this embodiment, the training process uses the registered PET images and CT images and their corresponding tumor segmentation labels as supervision signals, and continuously optimizes the model parameters through the gradient backpropagation mechanism to enable it to have good feature extraction and semantic discrimination capabilities.

[0057] In this embodiment, the registered PET and CT images were constructed using the publicly available medical imaging challenge dataset HECKTOR2022 (Head and Neck Tumor Segmentation and Outcome Prediction 2022). This dataset contains PET / CT bimodal medical images and tumor segmentation labels from head and neck tumor patients from multiple centers, along with accompanying clinical information and survival prognostic indicators. All images are three-dimensional volumetric data with inconsistent resolution and size, requiring uniform preprocessing.

[0058] Due to the complexity of data preparation and standardization processes (including resampling, cropping, registration, normalization, and label format standardization), all experimental data were processed using this standardized process to ensure the reproducibility of this invention. After preprocessing, all patients' CT images, PET images, and corresponding label images were cropped to a uniform voxel size of 144px × 144px × 144px to fit the input requirements of the proposed 3D convolutional neural network model. The preprocessed dataset served as the basis for subsequent model training and evaluation, ensuring the standardization and consistency of the experiment.

[0059] In S2, a combined loss function is used for optimization during the training of the DE-LS-UNet model to simultaneously balance the class imbalance problem and the fineness of segmentation contour boundaries. The specific expression is: In the formula, The Dice loss measures the degree of overlap between the predicted mask and the ground truth label. Let cross-entropy be the loss function. For weight hyperparameters, ; In the formula, For the model in the first i The probability value predicted at the individual element. For the first i The true label of individual factors To prevent smoothing factors with a denominator of zero; In the formula, voxels i Category C The predicted probability, This is a real label.

[0060] In this embodiment, the core idea behind training the DE-LS-UNet model is that the Dice loss has excellent region overlap optimization properties, making it suitable for scenarios with severe foreground-background imbalance in medical images; while the cross-entropy loss is more stable in class discrimination and voxel-level classification. Combining the two can balance the model's needs for segmentation accuracy and training stability, improving the network's ability to discriminate target regions with blurred boundaries and complex shapes.

[0061] At the start of training, preprocessed head and neck tumor training samples were used as network input. The input data included registered PET and CT images, which were stitched together in the channel dimension to form bimodal volumetric data. Each training sample had a uniform three-dimensional size specification to ensure consistency of network input dimensions.

[0062] During training, the Adam optimizer is used for iterative parameter updates, with the initial learning rate set to [value missing]. The model is dynamically adjusted based on validation set performance. Furthermore, to enhance generalization ability, data augmentation strategies such as random flipping and rotation are introduced during training to expand sample diversity. During training, each input sample undergoes forward propagation through an improved DE-LS-UNet network. The network first extracts modal features through independent encoders for the PET and CT branches, respectively. Then, in the fusion stage, a neighborhood-aware attention module enhances local perceptual representation, and a skip attention gating module suppresses non-target redundant feature information at skip connections. Finally, the decoder restores the spatial resolution and outputs the predicted tumor segmentation mask. Loss is calculated by comparing the predicted results from forward propagation with the ground truth labels, and the model parameters are updated and optimized based on backpropagation of the error using the combined loss function. Key metrics such as the validation set Dice coefficient, Precision, and Recall are continuously monitored during training to prevent overfitting, and the best-performing model weights are selected and saved.

[0063] In S2, after obtaining the trained DE-LS-UNet model, the new PET and CT images are input into the trained DE-LS-UNet model to obtain the predicted tumor segmentation mask. The tumor segmentation mask represents the spatial distribution of the tumor region predicted by the model, usually a voxel-level 0-1 mask. This mask can not only be used for clinical auxiliary diagnosis and target delineation, but also serve as the basic input for subsequent radiomics analysis and survival prognosis modeling, providing clinicians with quantitative information on tumor location, morphology and distribution.

[0064] S3 includes the following steps: S31. Based on the predicted tumor segmentation mask, extract radiomics features from the corresponding CT and PET images, including intensity statistical features, shape geometric features, texture features and high-level features; In this embodiment, the method for extracting radiomics features is as follows: the mask region predicted by the model is aligned with the original image space as a region of interest (ROI), and the open-source tool PyRadiomics is used to automatically extract multidimensional radiomics features within the region. The extracted features mainly include: Intensity statistical features (First-order statistics): such as mean, maximum, skewness, kurtosis, etc., reflect the basic distribution characteristics of tumor tissue in the gray space of the image; Shape features include volume, surface area, sphericity, aspect ratio, compactness, etc., which reflect the tumor boundary and spatial morphology. Texture features: derived from gray-level co-occurrence matrix (GLCM), gray-level run-length matrix (GLRLM), gray-level region matrix (GLSZM), etc., are used to characterize the texture consistency, directionality and complexity inside tumor tissue; Advanced features (Transformed features): Features extracted after image filtering (such as LoG, wavelet, LBP) to enhance local texture representation.

[0065] The aforementioned features were extracted independently from CT and PET images to ensure the preservation of unique spatial structure and metabolic activity characteristics in multimodal images. A set of features was generated for each patient from both CT and PET images, which were subsequently matched and stitched together using a unified patient identifier to form a patient-level high-dimensional radiomics feature vector.

[0066] Furthermore, to ensure consistency and comparability of features during feature extraction, a fixed parameter template was used for configuration, and a unified 3D extraction method was employed. The resulting feature table was saved in CSV format for subsequent modeling and analysis. The entire process ensured feature standardization, automation, and repeatability.

[0067] S32. Preprocess the radiomics features extracted from CT and PET images to generate CT feature vectors and PET feature vectors. Then, concatenate the CT and PET feature vectors of the same patient along the feature dimension to generate multimodal radiomics features. In this embodiment, CT images and PET images reflect anatomical structures and metabolic activities, respectively, and the features extracted from them are significantly complementary. To fully utilize the information from both modalities, the extracted omics features need to be fused and integrated: Preprocessing stage: Missing values ​​are filled and Z-scores are standardized for CT and PET features respectively to ensure that the numerical scale of each dimension of features is consistent and to avoid bias caused by distribution differences during model training.

[0068] Feature-level concatenation: This method concatenates the CT and PET feature vectors of the same patient along the feature dimension to form a unified image feature representation, fully integrating morphological and functional information. This fusion method preserves the differences between the original modalities while introducing interactive information, helping the model learn deeper-level correlations between multimodal features.

[0069] S33. Process the patient's clinical information to generate clinical features, and then integrate the clinical features with multimodal radiomics features in terms of feature dimensions to generate unified multi-source patient features.

[0070] In this embodiment, in the process of constructing the multimodal feature matrix, in addition to radiomics features, the patient's individual clinical characteristics also play a crucial role and are important factors affecting prognostic risk assessment. To improve the model's ability to discriminate in survival prediction tasks, these two types of heterogeneous features need to be effectively fused.

[0071] The clinical features used in this embodiment may include, but are not limited to, age, gender, weight, past medical history, viral marker status, and treatment methods. Past medical history includes factors such as smoking and alcohol consumption, and treatment methods include whether surgery or chemotherapy was performed. Continuous variables are filled with missing values ​​using the median and then standardized and normalized to ensure their numerical scale is consistent with the imaging features. Discrete variables are one-hot encoded to convert them into a learnable numerical form, and the "missing" information is explicitly retained as an independent category to improve the robustness of the model.

[0072] The processed clinical feature vectors are concatenated and integrated with multimodal radiomics features to generate a final unified patient feature representation. This fused feature matrix comprehensively represents the structural, metabolic, pathological, and treatment-responsive information of an individual patient and is a key input in the survival prognosis modeling stage.

[0073] In addition, to improve modeling efficiency and generalization ability, feature selection strategies can be introduced into the fused feature matrix, such as methods based on variance threshold, mutual information, maximum correlation minimum redundancy (mRMR), Lasso regularization, etc., to remove redundant dimensions and retain only key factors that are significantly related to recurrence-free survival time, so as to optimize the performance of subsequent prediction models.

[0074] like Figure 6 As shown in Figure S4, the survival analysis model consists of an input layer, a hidden layer, and an output layer connected in sequence. The input layer inputs multi-source feature representations; the hidden layer consists of several layers, each with the same structure, including a ReLU activation function layer, a Dropout layer, and a linear layer connected in sequence; the output layer outputs a real-valued risk score.

[0075] In this embodiment, the input layer's input dimension is the feature dimension of the patient's multi-source features. The hidden layer is used to model the nonlinear relationship between features. The ReLU activation function layer and the Dropout layer are interconnected to enhance expressive power and prevent overfitting. The linear layer is used to extract higher-order nonlinear representations of features layer by layer. The output layer is a single-neuron linear layer that outputs a single scalar risk score. This represents the relative risk of recurrence for each individual.

[0076] In the formula, For each patient, multi-source feature expression, It is the ReLU activation function. For Dropout layer, It is a linear layer.

[0077] In S4, the negative logarithmic loss function is used for optimization during the training of the survival analysis model to effectively handle censored data. The specific expression of the negative logarithmic loss function is as follows: In the formula, For event indicator, 1 indicates a patient relapse, 0 indicates censorship. For follow-up time, In order to be in The sample index set that is still in the risk concentration at all times. To indicate the first j Individual sample risk score exponential form, A single scalar risk score output by the output layer; In this embodiment, the training objective of the model is based on the negative log loss defined by the Cox partial likelihood function. This loss function can effectively handle censored data. The model does not need to predict a specific survival time, but only needs to fit the relative risk ranking, which has good robustness and interpretability.

[0078] During the training phase, patient multi-source features were divided into training and testing sets and standardized. A negative logarithmic loss function was used to optimize network parameters during training, ensuring that the predicted risk scores accurately reflected the patients' true survival status and order. The Adam optimizer was used for parameter updates during training, and appropriate learning rates, batch sizes, and training epochs were set to achieve convergence.

[0079] After the model training is completed, the features of all patients are input into the trained DeepSurv model to obtain the predicted risk score for each individual. Based on the relative level of the risk score, patients can be divided into low-risk and high-risk groups. Furthermore, combined with real RFS label information, the survival time distribution is visualized through Kaplan-Meier curves. The predictive performance of the model is comprehensively evaluated by combining multiple evaluation indicators such as the consistency index (C-index) and the Brier score, thus verifying the effectiveness of the model in distinguishing recurrence risk.

[0080] like Figure 7 The graph shows the changes in the consistency index and Brill score during training. As observed during model training, the consistency index continuously increased with each training round, rising from an initial 0.6251 to a peak of 0.8534, ultimately reaching its best performance in round 58 (training was prematurely terminated in round 59 due to numerical instability). The final consistency index C-index of 0.8257 was obtained in the full data evaluation, indicating strong consistency between the predicted results and the actual recurrence time. During this period, the Brill score continuously decreased to a minimum of 0.1860, showing that the model has good probability calibration ability, consistent with the negative correlation between risk scores and recurrence events, indicating that the model has a strong ability to discriminate high-risk samples.

[0081] To further validate the model's effectiveness in patient risk stratification, this invention outputs typical Kaplan-Meier survival curves, such as... Figure 8As shown in the figure, the predicted risk score is used to divide patients into high-risk and low-risk groups. The survival curves of the two groups show a clear separation trend, with the relapse-free survival time of patients in the low-risk group being significantly longer than that of the high-risk group. This figure demonstrates that the model has a strong ability to stratify patients and differentiate prognoses in clinical practice.

[0082] Furthermore, this invention also plotted a scatter plot between survival time and predicted risk score, such as... Figure 9 As shown in the figure, different colors indicate whether a relapse event occurred (Relapse=1). The figure reveals that patients with shorter survival times generally have higher risk scores, further validating the model's effectiveness in identifying high-risk individuals. Risk scores are concentrated in the early survival stages and gradually become sparser, exhibiting good temporal correlation characteristics.

[0083] In summary, based on the feature extraction and fusion mechanism proposed in this invention, combined with the DeepSurv survival modeling strategy, the final survival prediction model performs excellently on multiple evaluation indicators, and can accurately predict and stratify the risk of patient recurrence, providing a reliable tool to support tumor prognosis decision-making.

[0084] In this embodiment, to verify the effectiveness of the method of the present invention, the following experimental data are provided: (1) In order to further verify the effectiveness of the design of each key module in this invention, a series of systematic ablation experiments were carried out. Under the unified data preprocessing process, model training settings and evaluation criteria, the neighborhood perception attention module and the jump attention gating module were gradually introduced, and their integration effect at different positions was explored, so as to comprehensively evaluate the actual contribution of each structural improvement to the model performance.

[0085] This ablation study was conducted using Python 3.8, PyTorch 2.3.0, and CUDA 11.8, employing a 16GB NVIDIA 4060TI GPU. The batch size was set to 3, and the number of training epochs was 300. The Adam optimizer was used, and a cosine annealing learning rate scheduling strategy was introduced during training. The loss function used in the experiment was the same as in step two (see step two for details). The performance of each experiment was comprehensively evaluated using common metrics such as Dice coefficient (DSC), Hausdorff Distance (HD), Precision, and Recall. The number of parameters of the corresponding models was also statistically analyzed to fully reflect the trade-off between accuracy and complexity.

[0086] All training and evaluation were conducted under the same settings to ensure the fairness and reproducibility of the comparative experiments. The results of the ablation experiments are shown in Table 1.

[0087] Table 1 Ablation Experiment Results Experimental data shows that the baseline model (single-branch, no attention) performed the lowest in DSC, Precision, and Recall metrics, at 0.7927, 0.7817, and 0.8171 respectively, while HD was the highest at 8.8219. This indicates that the original 3D U-Net structure has limited fusion capabilities when dealing with multimodal data. Without adding any attention module, introducing only an independent dual-branch encoder improved DSC to 0.8042, demonstrating that independent modeling of CT and PET helps enhance feature representation capabilities. Although HD increased slightly, it remained within an acceptable range.

[0088] In a single-branch structure, the performance improvements from adding LAA and SAG differ. The DSC after introducing LAA is 0.7935, a relatively small improvement, indicating that single spatial attention has limited effectiveness when feature representation is constrained. However, introducing SAG improves the DSC to 0.8059 and the recall to 0.8229, demonstrating that the guided gating mechanism is more sensitive to segmentation boundaries and small structures. In a dual-branch structure, introducing LAA or SAG alone also yields further improvements, with DSCs reaching 0.8058 and 0.8025 respectively, indicating that both modules demonstrate their advantages based on independent feature encoding.

[0089] By simultaneously introducing LAA and SAG into the single-branch structure, DSC increased to 0.8093 and Precision reached 0.8085, demonstrating the synergistic enhancement effect of module combination. The optimal structure appeared in the "dual-branch + LAA + SAG" combination, namely the complete improved network DE-LS-UNet proposed in this invention, which achieved the best results on all indicators: DSC=0.8139, Precision=0.8113, Recall=0.8290, and HD decreased to the lowest value of 7.2101, fully demonstrating the comprehensive performance advantages of this invention.

[0090] In summary, the dual-branch structure provides a more reasonable foundation for modal separation modeling, the LAA module enhances spatial information perception, and the SAG module optimizes the cross-layer feature guidance path. The modules are structurally complementary and can significantly improve the model's segmentation accuracy and boundary robustness when used together, fully validating the scientific validity and practical value of the proposed improved structure.

[0091] (2) To fully verify the effectiveness of the proposed DE-LS-UNet model in actual tumor image processing tasks, this invention uses the HECKTOR2022 dataset as the experimental basis and compares its performance with several existing mainstream 3D medical image segmentation networks to systematically evaluate the segmentation performance of the proposed model under multimodal input conditions. All experiments are based on a unified preprocessing procedure and training strategy, and are conducted under the same CT and PET input conditions to ensure the fairness and reproducibility of the comparison results.

[0092] The comparative experiments focused specifically on the model's recognition ability in the head and neck tumor region, and the results were evaluated using multiple metrics commonly used in medical image segmentation. These experiments visually demonstrate the performance improvements of the proposed dual-branch coding structure, local perceptual attention mechanism, and skip-gating fusion strategy on real-world datasets. The experimental results are shown in Table 2.

[0093] Table 2 Comparison of experimental results Based on the above comparative experimental results, on the HECKTOR2022 dataset, the improved model DE-LS-UNet proposed in this invention was systematically compared and evaluated with the current mainstream 3D medical image segmentation networks, covering four key performance indicators: DSC, Precision, Recall, and Hausdorff distance.

[0094] In terms of Dice similarity coefficient, DE-LS-UNet ranks first with a score of 0.8139, slightly higher than TransUNet-3D (0.8127) and SwimUNet-3D (0.8116), and significantly better than traditional networks such as 3D U-Net (0.7927) and V-Net (0.7993), showing stronger overall segmentation accuracy.

[0095] In terms of precision, TransUNet-3D performed best (0.8174), but DE-LS-UNet followed closely with 0.8113 and made up for the gap in recall.

[0096] In terms of recall, SwimUNet-3D (0.8293) is slightly higher than DE-LS-UNet (0.8290), but the two are almost equal, and it is far superior to V-Net and 3D U-Net. This indicates that the model of this invention has a very strong ability to capture complete tumor regions with an extremely low false negative rate.

[0097] Especially in the Hausdorff distance, DE-LS-UNet significantly outperformed other comparative models with a score of 7.2101, such as TransUNet-3D at 8.5037 and SwimUNet-3D at 8.7149, indicating that it has higher accuracy in tumor boundary localization and its contour prediction is more realistic.

[0098] In summary, DE-LS-UNet achieves a balance between precision and recall while demonstrating superior segmentation margins and overall accuracy, proving the effectiveness and advancement of its network architecture design. This model achieves optimal or near-optimal performance across multiple key metrics, demonstrating strong clinical applicability and engineering application value.

[0099] In the description of this invention, it should be understood that the terms "center," "thickness," "upper," "lower," "horizontal," "top," "bottom," "inner," "outer," and "radial," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include one or more of that feature.

Claims

1. A multimodal head and neck tumor segmentation and survival prognosis prediction method, characterized in that, The method comprises the following steps: S1, constructing a DE-LS-UNet model, the specific process is: based on a 3D U-Net model, a neighborhood perception attention module is arranged in the encoder part of the fourth layer and the fifth layer, the input of each layer of the neighborhood perception attention module is connected with the output of the three-dimensional maximum pooling operation of the previous layer, and the output of each layer of the neighborhood perception attention module is connected with the input of the convolution module of the encoder part of the layer; a skip attention gating module is arranged after the skip connection path of the decoder part of the first layer to the fourth layer, the input of each layer of the skip attention gating module is connected with the output of the skip connection path and the output of the convolution module of the decoder part of the next layer respectively, and the output of each layer of the skip attention gating module is connected with the input of the convolution module of the decoder part of the layer; S2, inputting the registered PET image and CT image into the DE-LS-UNet model, training the DE-LS-UNet model, obtaining a trained DE-LS-UNet model, inputting a new PET image and CT image into the trained DE-LS-UNet model, and obtaining a predicted tumor segmentation mask; S3, extracting multi-modal image feature according to the predicted tumor segmentation mask, and fusing clinical features through the multi-modal image feature to generate patient multi-source features; S4, inputting the patient multi-source features into a survival analysis model, training the survival analysis model, obtaining a trained survival analysis model, inputting the patient features into the trained survival analysis model, and obtaining a prognosis prediction result.

2. The multimodality head and neck tumor segmentation and survival prognosis prediction method of claim 1, wherein, In S1, the working process of the DE-LS-UNet model is specifically: S11, inputting the CT image and the PET image into the DE-LS-UNet model, performing convolution operation on the CT image and the PET image to generate an intermediate feature map, wherein the convolution operation is specifically two times of convolution module processing, the convolution module processing is specifically 3*3*3 three-dimensional convolution operation, batch normalization and PReLU activation function processing in sequence, three-dimensional maximum pooling operation is performed on the intermediate feature map to obtain a CT feature map and a PET feature map; S12, performing convolution operation on the CT feature map and the PET feature map to generate an intermediate feature map, and performing channel dimension splicing and fusion on the two obtained intermediate feature maps to obtain a first feature map; S13, performing convolution operation, three-dimensional maximum pooling operation and neighborhood perception attention module processing on the first feature map for two times in succession to obtain a second feature map, and an intermediate feature map is generated each time the convolution operation is performed; S14, performing convolution operation on the second feature map to obtain a third feature map, performing three-dimensional upsampling, skip attention gating module processing and convolution operation on the third feature map for four times in succession to obtain a fourth feature map, and after each three-dimensional upsampling, the intermediate feature map of the layer is input into the skip attention gating module through the skip connection path; S15, performing 1*1*1 three-dimensional convolution operation on the fourth feature map to obtain the output result of the DE-LS-UNet model.

3. The multimodality head and neck tumor segmentation and survival prognosis prediction method of claim 2, wherein, In S1, the working process of the neighborhood perception attention module is specifically: A1, the input features of the neighborhood perception attention module are subjected to a three-dimensional convolution operation to generate query features and value features, the value features are subjected to a neighborhood expansion operation in a three-dimensional space to extract neighborhood feature subsets; A2, similarity scores are calculated by element-wise multiplication of each neighborhood feature subset and the query features corresponding to the position, and all the obtained similarity scores are normalized in the domain dimension to obtain a domain weight distribution; A3, the initial output features are calculated based on the domain weight distribution and the neighborhood feature subsets, and a 1x1x1 three-dimensional convolution is used for projection on the initial output features, and the projection result is added to the input feature residual which is cropped to the same size as the projection result to obtain the output features of the neighborhood perception attention module.

4. The multimodality head and neck tumor segmentation and survival prognosis prediction method of claim 2, wherein, In S1, the working process of the skip attention gating module is as follows: B1, based on the high-level semantic feature map obtained by three-dimensional upsampling and the intermediate feature map output by the skip connection channel, the high-level semantic feature map and the intermediate feature map are subjected to 1x1x1 three-dimensional convolution to reduce the dimension to a mapping feature with the same number of intermediate channels, and the two mapping features are added point by point and then activated by a ReLU function to obtain an activated feature map; B2, the activated feature map is subjected to 1x1x1 three-dimensional convolution and then activated by a Sigmoid function to obtain a spatial attention weight shared between channels, the intermediate feature map is subjected to element-wise multiplication operation through the spatial attention weight, and the element-wise multiplication result is concatenated with the high-level semantic feature map in the channel to obtain the output feature of the skip attention gating module.

5. The multimodality head and neck tumor segmentation and survival prognosis prediction method of claim 1, wherein, In S2, a combined loss function is used to optimize the training process of the DE-LS-UNet model to balance the class imbalance problem and the fineness of the segmentation contour boundary at the same time, and the expression of the combined loss function is Specifically, the expression of the combined loss function is: wherein is the Dice loss, which measures the degree of overlap between the predicted mask and the true label, is the cross-entropy loss function, is the weight hyperparameter, ; wherein is the probability value predicted by the model at the i voxel, is the true label of the i voxel, is a smoothing factor to prevent the denominator from being zero; wherein is a voxel i belongs to a class C predicted probability, is the true label.

6. The multimodality head and neck tumor segmentation and survival prognosis prediction method of claim 1, wherein, S3 includes the following steps: S31, based on the predicted tumor segmentation mask, imageomic features are extracted from the corresponding CT image and PET image, including intensity statistical features, shape geometric features, texture features and high-level features; S32, the imageomic features extracted from the CT image and the PET image are preprocessed to generate a CT feature vector and a PET feature vector, and the CT feature vector and the PET feature vector of the same patient are concatenated in the feature dimension to generate a multi-modal imageomic feature; S33, the clinical information of the patient is processed to generate a clinical feature, and the clinical feature and the multi-modal imageomic feature are concatenated and integrated in the feature dimension to generate a unified patient multi-source feature.

7. The multimodality head and neck tumor segmentation and survival prognosis prediction method of claim 1, wherein, In S4, the survival analysis model includes an input layer, a hidden layer and an output layer connected in turn, and the input layer inputs the multi-source feature expression; the hidden layer includes several layers, each layer has the same structure and includes a ReLU activation function layer, a Dropout layer and a linear layer connected in turn; the output layer outputs a real number type risk score.

8. The multimodality head and neck tumor segmentation and survival prognosis prediction method of claim 7, wherein, In S4, a negative log loss function is used for optimization in the training process of the survival analysis model to effectively handle censored data, and the expression of the negative log loss function is as follows: wherein, is an event indicator, 1 indicates a patient relapse, 0 indicates a censoring, is a follow-up time, is a set of sample indices still in the risk set at time, is an exponential form representing the risk score of the j sample, is a single scalar risk score output by the output layer;​ wherein, is the multi-source feature representation for each patient, is the ReLU activation function, is the Dropout layer, is the linear layer.

Images