Head and neck tumor image segmentation method based on combined analysis of CT and PET

Abstract

The invention discloses a head and neck tumor image segmentation method based on combined analysis of CT and PET, and the method comprises the steps: obtaining a head and neck tumor data set, and carrying out the preprocessing of the head and neck tumor data set; constructing a parallel common attention mechanism module; an improved UNet3D network model is constructed; optimizing the improved UNet3D network model by adopting a combined loss function; and inputting the preprocessed head and neck tumor data set into the optimized and improved UNet3D network model, and outputting a segmentation task result of the bimodal image. According to the method, a parallel common attention mechanism and a multi-scale feature fusion technology are introduced, multi-modal image information is efficiently combined, features of the tumor under different scales are captured, accurate segmentation is carried out on the features, the precision and efficiency of head and neck tumor segmentation are remarkably improved, the limitation that a single-modal image is difficult to comprehensively reflect the tumor features is overcome, and the accuracy and efficiency of head and neck tumor segmentation are improved. And the problem of scale diversity of the tumor in the image is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of image segmentation, and in particular relates to a head and neck tumor image segmentation method based on CT and PET combined analysis. Background Art

[0002] Head and neck tumors are one of the most common types of malignant tumors in the world, including nasopharyngeal carcinoma, oral cancer, and laryngeal cancer. Due to the complex anatomical structure of the head and neck and the blurred tumor boundaries, clinical diagnosis and treatment are difficult. Accurate tumor segmentation is of great significance for the formulation of treatment plans, surgical planning, and radiotherapy. At present, the tumor segmentation methods widely used in clinical practice mostly rely on manual annotation by doctors or rule-based semi-automatic algorithms. However, these traditional methods have great limitations in complex anatomical areas and are difficult to cope with the irregularity of tumor morphology and uncertainty of boundaries. Therefore, the development of automated and intelligent tumor segmentation methods to improve segmentation efficiency and accuracy has become one of the important directions of current medical imaging research.

[0003] Multimodal imaging technology shows great potential in the diagnosis and treatment of head and neck tumors. Computed Tomography (CT) provides high-resolution anatomical information, while positron emission tomography (PET) can reflect the metabolic activity of tissues. Joint analysis of these two different modal imaging data can more comprehensively reflect the biological characteristics of the tumor and help improve the accuracy of tumor segmentation. However, due to the differences in spatial resolution and image features between CT and PET, how to achieve efficient fusion of the two modal data is still one of the difficulties of segmentation technology. Traditional single-modal segmentation methods often perform poorly when processing such multimodal data. Therefore, the use of deep learning models that can effectively combine multimodal information, especially segmentation methods based on convolutional neural networks (CNN), is the key to improving segmentation performance.

[0004] In recent years, deep learning technology, especially convolutional neural networks (CNNs), has made breakthrough progress in medical image segmentation tasks. Compared with traditional methods, deep learning automatically learns image features in a data-driven manner, greatly reducing the burden of manually designing features, and especially showing excellent performance when processing complex images. Three-dimensional convolutional neural networks (3D-CNNs) have shown powerful capabilities in learning three-dimensional spatial features of medical images, and can make full use of the information of three-dimensional data to significantly improve the accuracy of tumor segmentation. In particular, the attention mechanism introduced in recent years can help the model automatically focus on key areas in the image, while improving the accuracy of tumor edge segmentation and reducing background interference. In addition, multi-scale fusion technology further enhances the model's sensitivity to tumor morphology by combining features of different scales, making it more robust in complex tumor segmentation tasks.

[0005] The development of automated medical image segmentation technology has provided new ideas and methods for tumor segmentation. The automated segmentation system based on deep learning can not only reduce the time cost of manual operation, but also maintain a high segmentation efficiency when facing a large amount of image data. In the joint analysis of multimodal data, automated segmentation methods have shown great potential, especially in the segmentation tasks of complex anatomical areas such as head and neck tumors. By integrating the multimodal information of CT and PET, the automated segmentation system can significantly improve the accuracy and robustness of segmentation. However, despite the excellent performance of deep learning models in segmentation tasks, they still face some challenges in clinical applications, including heterogeneity of image data, imaging differences between devices, artifacts and noise interference. Therefore, how to enhance the generalization ability of the model and ensure its stable performance under different patient and device conditions is an important challenge in the current clinical application of automated segmentation technology.

[0006] In the clinical treatment of head and neck tumors, traditional manual segmentation methods rely on the doctor's experience to operate. This segmentation method is time-consuming and laborious, which not only increases the doctor's workload, but is also easily affected by subjective judgment, resulting in significant differences in results between different doctors. In addition, the anatomical structure of the head and neck region is complex, and the boundary between the tumor and the surrounding normal tissue is usually blurred. Traditional manual segmentation methods have difficulty accurately distinguishing these subtle differences, resulting in insufficient tumor segmentation accuracy and stability. Especially when it is necessary to carefully mark complex tumor boundaries, manual operations are not only inefficient, but also difficult to guarantee accuracy, which directly affects subsequent treatment plans.

[0007] The application of single-modality images (such as CT or PET) in medical segmentation also has significant limitations. CT images provide anatomical structural information, while PET reflects metabolic function information. Relying solely on images of any one modality for tumor segmentation cannot fully reflect the characteristics of the tumor. The incompleteness of this information often makes it difficult to accurately identify tumor boundaries, especially when the tumor morphology is complex or the boundaries are blurred. Single-modality images cannot provide sufficient contrast and the segmentation results are less accurate. In addition, traditional deep learning segmentation models mostly process single-modality data, and existing technologies do not make sufficient joint use of multi-modality images, resulting in the model being unable to effectively capture the complementary information in CT and PET images, restricting the improvement of segmentation performance.

[0008] The introduction of attention mechanism in medical image segmentation has been a research hotspot in recent years. However, most of the existing attention mechanisms only extract features for a single modality or a single scale, which makes it difficult to fully capture the multi-scale information of the tumor. When dealing with complex and changeable targets such as head and neck tumors, the existing attention mechanism cannot accurately locate important feature areas, affecting the accuracy of segmentation. Although multi-scale fusion can effectively enhance the network's ability to capture tumor features at different resolutions, the existing multi-scale fusion strategies often face the problem of feature information loss or poor fusion effect, resulting in poor performance of the segmentation model when dealing with complex lesions. In addition, although the deep learning model has achieved automated segmentation to a certain extent, the generalization ability of the model and its sensitivity to clinical noise are still the main bottlenecks that make it difficult to promote. Summary of the invention

[0009] The purpose of the present invention is to address the above-mentioned deficiencies in the prior art and to provide a head and neck tumor image segmentation method based on joint analysis of CT and PET, so as to solve the problem that the prior art does not make sufficient joint use of multimodal images, resulting in the inability of the model to effectively capture the complementary information in CT and PET images, thereby restricting the improvement of segmentation performance.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] A head and neck tumor image segmentation method based on CT and PET joint analysis includes the following steps:

[0012] S1, obtaining a head and neck tumor dataset including CT images, PET images and segmented label images, and preprocessing the head and neck tumor dataset;

[0013] S2. Build a parallel co-attention mechanism module;

[0014] S3, inserting several parallel co-attention mechanism modules into the backbone network of the UNet3D network model to construct an improved UNet3D network model;

[0015] S4, using the combined loss function to optimize the improved UNet3D network model;

[0016] S5. Input the preprocessed head and neck tumor dataset into the optimized improved UNet3D network model, and output the segmentation task results of the dual-modal image.

[0017] Furthermore, the neck tumor dataset is preprocessed in S1, including the following steps:

[0018] S11, resampling the CT image and the PET image using a triple linear interpolation method, and resampling the label image using a nearest neighbor interpolation method; when sampling the CT image, the PET image, and the label image, adjusting the size of the resampled image based on the spatial direction, the starting origin, and the coordinate system of the CT image, until the voxel spacing of the CT image, the PET image, and the label image is consistent and the spatial coordinate systems are aligned;

[0019] S12, detecting the spatial position of the tumor in the label image, determining the bounding box where the tumor is located, calculating the center point of the tumor according to the position of the bounding box, and cropping the CT image, PET image, and label image to a size of 144px×144px×144px based on the center point;

[0020] S13, performing z-score normalization processing on the PET image, and performing grayscale value truncation and maximum-minimum normalization processing on the CT image in sequence.

[0021] Further, S2 includes the following sub-steps:

[0022] S21, inputting the CT image features and the PET image features into the SE module respectively, and outputting the feature weighted vectors of the CT image and the PET image respectively;

[0023] S22, calculating an affinity matrix according to the feature weighted vectors of the CT image and the PET image;

[0024] S23, calculating the co-attention weights of the CT image features and the PET image features respectively according to the affinity matrix;

[0025] S24. Calculate the weighted sum of the CT image features and the PET image features according to their co-attention weights, respectively, and concatenate the weighted sum of the CT image features and the PET image features to obtain a fused co-attention feature map.

[0026] Furthermore, in S21, the feature weighted vectors of the CT image and the PET image are output, including:

[0027]

[0028]

[0029] Among them, F ct and F pet are CT image features and PET image features respectively; X is the global feature; GAP is the global average pooling; T is the feature vector output by the fully connected layer; FC is the fully connected layer; σ1 is the ReLU activation function, σ2 is the sigmoid activation function; se is the weight; ⊙ is the element-by-element multiplication operation; Q and V are the feature weighted vectors of CT image features and PET image features respectively.

[0030] Further, in S22, an affinity matrix is ​​calculated, including:

[0031] Affinity=tanh(W b (Q)·W b (V)

[0032] Among them, Affinity is the affinity matrix; tanh is the activation function; W b is the weight matrix.

[0033] Furthermore, in S23, the co-attention weights of the CT image features and the PET image features are calculated respectively, including:

[0034] H v =tanh(W v (V)+W q (Q) Affinity

[0035] H q =tanh(W q (Q)+W v (V) Affinity

[0036] a v =softmax(W h ·H v )

[0037] a q =softmax(W h ·H q )

[0038] Among them, H v , H q are the modality enhancement feature vectors of PET image and CT image respectively; W v , W q , W h are weight parameters respectively; a v and aq are the co-attention weights of PET image features and CT image features, respectively.

[0039] Furthermore, in S24, the fused co-attention feature map is obtained, including:

[0040] F′ ct =∑a q Q

[0041] F′ pet =∑a v ·V

[0042] F fusion =Concat(F′ ct ,F′ pet )

[0043] Among them, F′ ct and F′ pet is the output feature after weighting CT image features and PET image features; Concat is the concatenation operation; F fusion It is the fused co-attention feature map.

[0044] Furthermore, S3 includes the following sub-steps:

[0045] S31, inputting the CT image and the PET image into two independent UNet encoder stages, respectively, outputting preliminary features q1, q2, q3, and q4 of the CT image, and outputting preliminary features v1, v2, v3, and v4 of the PET image;

[0046] S32, insert a parallel co-attention mechanism module into the back end of each encoder stage of the backbone network of UNet3D;

[0047] The original CT image and PET image data are input into the first parallel co-attention mechanism module, and the fused bimodal feature F0 is output;

[0048] The preliminary features q1 of the CT image and v1 of the PET image obtained in the first stage are input into the second parallel co-attention mechanism module, and the fused bimodal features F1 are output;

[0049] The preliminary features q2 of the CT image and v2 of the PET image obtained in the second stage are input into the third parallel co-attention mechanism module, and the fused bimodal features F2 are output;

[0050] The preliminary features q3 of the CT image and v3 of the PET image obtained in the third stage are input into the fourth parallel co-attention mechanism module, and the fused bimodal features F3 are output;

[0051] The preliminary features q4 of the CT image and v4 of the PET image obtained in the fourth stage are input into the fifth parallel co-attention mechanism module, and the fused bimodal features F4 are output;

[0052] S33, splicing the dual-modal feature F0 and the original CT image and PET image data to form a feature vector with 4 channels, and inputting it into the encoder module of the first stage, and the number of channels becomes 16 after feature extraction by the encoder module;

[0053] The concatenated feature map output by the encoder module in the first stage is concatenated with F1 and input to the encoder module in the second stage. After feature extraction by the encoder module, the number of channels becomes 32;

[0054] The concatenated feature map output by the second stage encoder module is concatenated with F2 and input into the third stage encoder module. After feature extraction by the encoder module, the number of channels becomes 64;

[0055] The concatenated feature map output by the encoder module in the third stage is concatenated with F3 and input into the encoder module in the fourth stage. After feature extraction by the encoder module, the number of channels becomes 128;

[0056] The concatenated feature map output by the fourth stage encoder module is concatenated with F4 and input into the fifth stage encoder module. After feature extraction by the encoder module, the number of channels becomes 256.

[0057] S34, in the decoder part, the output of the encoder module of the fifth stage is deconvolved to restore the number of channels from 256 to 128, and the output is concatenated with the feature map of the encoder module of the fourth stage, and features are extracted through convolution operation;

[0058] The output of the decoder module in the fourth stage is deconvolved to restore the number of channels from 128 to 64, and the output is concatenated with the feature map of the encoder module in the third stage, and features are extracted through convolution operations;

[0059] The output of the decoder module in the third stage is deconvolved to restore the number of channels from 64 to 32, and the output is concatenated with the feature map of the encoder module in the second stage, and features are extracted through convolution operations;

[0060] The output of the decoder module of the second stage is deconvolved to restore the number of channels from 32 to 16, and the output is concatenated with the feature map of the encoder module of the first stage, and features are extracted through convolution operation;

[0061] The output of the decoder module in the first stage is passed through a 1x1x1 convolutional layer to reduce the number of channels from 16 to 2, and the segmentation task result is output.

[0062] Furthermore, in S4, the combined loss function includes the Soft Dice Loss loss function and the weighted FocalLoss loss function, which is specifically:

[0063] Loss=Soft Dice Loss+0.5×Focal Loss

[0064] Focal Loss(p)=-α(1-p) γ log(p)

[0065] Among them, Loss is the combined loss function; p is the confidence of the model prediction; α and γ are parameters.

[0066] The head and neck tumor image segmentation method based on CT and PET joint analysis provided by the present invention has the following beneficial effects:

[0067] 1. By introducing the parallel co-attention mechanism and multi-scale feature fusion technology, the present invention can efficiently combine multimodal image information, capture the characteristics of tumors at different scales, and accurately segment them. The method of the present invention can not only improve the accuracy and efficiency of head and neck tumor segmentation, but also provide more reliable technical support for clinical diagnosis and treatment.

[0068] 2. The present invention combines deep learning with multimodal imaging technology, fully utilizes the anatomical structure information in CT images and the metabolic function information in PET images, overcomes the limitation that single-modal images are difficult to fully reflect tumor characteristics, and significantly improves the accuracy of segmentation results.

[0069] 3. The present invention introduces the Parallel Co-Attention (PCA) mechanism and multi-scale feature fusion technology. The attention mechanism can automatically focus on the important features of the tumor area and ignore irrelevant information, making the model more accurate in processing complex tumor morphology; multi-scale feature fusion ensures that the details of tumors of different sizes can be captured and processed, thus solving the problem of scale diversity of tumors in images.

[0070] 4. The present invention also provides a method for preprocessing CT and PET images to improve the quality and consistency of the model input images. The image preprocessing process includes resampling, cropping and normalizing the CT and PET data to ensure the alignment and clarity of multimodal images during joint analysis. Through this preprocessing method, the present invention can better ensure that the segmentation model still maintains a high performance when facing image data of different sources and different qualities.

[0071] 5. The present invention retains the detailed information of the multimodal features of the image: by resampling and aligning the coordinate system of the CT image, PET image and Label image, the spatial consistency of the multimodal image is ensured, and the details and structural information of the image are retained.

[0072] 6. The present invention improves the model's attention to the tumor area: the parallel co-attention mechanism (PCA) is used to deeply fuse CT and PET features, which enhances the model's comprehensive perception of image information of different modalities and improves the segmentation effect of tumor and pathological areas.

[0073] 7. The present invention enhances the feature expression capability: by introducing the PCA module in each encoding stage, cross-modal fusion of CT and PET image features is achieved, the model's ability to represent complex images is enhanced, and the segmentation accuracy is improved.

[0074] 8. This invention improves the stability and segmentation performance of model training: Combining the combined loss function of Soft Dice Loss and weighted Focal Loss effectively handles the problem of class imbalance, enhances the focus on difficult-to-classify samples, and further improves the model segmentation performance BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is the data preprocessing operation flow of the present invention.

[0076] Figure 2 This is a comparison diagram before and after the pretreatment of the present invention.

[0077] Figure 3 Schematic diagram of the PCA module structure of the present invention.

[0078] Figure 4 This is a schematic diagram of the SE module structure of the present invention.

[0079] Figure 5 This is the PCA-UNet network model architecture of the present invention.

[0080] Figure 6 Obtain v for the present invention i and q i Structure diagram of .

[0081] Figure 7 This is a comparison chart of the segmentation results of each model of the present invention.

[0082] Figure 8 The present invention is a flow chart of a head and neck tumor image segmentation method based on CT and PET joint analysis. DETAILED DESCRIPTION

[0083] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0084] Example 1

[0085] The head and neck tumor image segmentation method based on CT and PET joint analysis in this embodiment combines deep learning with multimodal imaging technology, fully utilizes the anatomical structure information in CT images and the metabolic function information in PET images, overcomes the limitation that single-modal images are difficult to fully reflect tumor characteristics, and significantly improves the accuracy of segmentation results. Figure 8 , which specifically includes the following:

[0086] Step S1, obtaining a head and neck tumor dataset including CT images, PET images and segmented label images (Label), and preprocessing the head and neck tumor dataset, which specifically includes the following sub-steps:

[0087] Step S11, resampling;

[0088] The spatial voxel spacing of the input CT image, PET image, and Label image is resampled to 1mm×1mm×1mm, and the spatial coordinates of the CT image are used as the unified benchmark of the three images to ensure that the voxel spacing of the three images is consistent and the spatial coordinate systems are aligned, so that they match accurately in space. The specific steps are as follows:

[0089] Firstly, the CT images and PET images are resampled by using the trilinear interpolation method to maintain the smoothness and continuity of the image details and ensure that the grayscale information and fine structure are not lost during the resampling process.

[0090] For label images, since they contain discrete category values, in order to avoid confusion of category values, the nearest neighbor interpolation method is used for resampling to ensure that the category values ​​remain unchanged after resampling and to avoid the introduction of non-integer intermediate values ​​in the interpolation process.

[0091] During the resampling process, the spatial direction, starting origin and coordinate system of the CT image are uniformly adopted, and the size of the resampled image is adjusted accordingly according to its original voxel spacing to ensure that the spatial resolution of the three is consistent after resampling and can be strictly aligned in the same spatial coordinate system.

[0092] Step S12, image cropping;

[0093] The resampled Label image is cropped. First, the pixels with a label value of 2 (lymph nodes) are relabeled as background value 0, and only the background (value 0) and tumor (value 1) are retained. Then, the spatial dimensions of the CT image, PET image, and Label image are cropped to reduce the image size and reduce the computational pressure of subsequent model training, while retaining the effective information area containing the tumor.

[0094] The specific operation steps are as follows: First, the spatial position of the tumor in the Label image is detected to determine the bounding box where the tumor is located. The center point of the tumor is calculated based on the position of the bounding box, and then the CT image, PET image, and Label image are cropped to a fixed size of 144px×144px×144px based on the center point to ensure that the tumor area is completely preserved. At the same time, the image boundaries are taken into account during the cropping process to ensure that the cropping operation does not exceed the actual size of the image. If the cropped area is smaller than the expected size, the starting point is adjusted appropriately to ensure that the final cropped image and label meet the predetermined size requirements.

[0095] Step S13, image normalization;

[0096] To perform z-score normalization on the PET image, first calculate the mean and standard deviation of the PET image, and then normalize each pixel according to the following formula, where z is the normalized value, x is the original data point, μ is the mean of the image, and σ is the standard deviation.

[0097]

[0098] Next, the grayscale value (Hounsfield units, HU) of the CT image is truncated to limit its value to the range of [-1024, 1024]. This can avoid the influence of extreme values ​​in the CT image on subsequent processing and retain the common medical image data range. Finally, the maximum-minimum normalization (Min-Max Normalization) is used to normalize the CT image and map its grayscale value to [-1, 1] to ensure the consistency of the image data range, which is helpful for subsequent neural network learning and processing. The maximum-minimum normalization can be expressed by the following formula (2), where x is the original pixel value, x′ is the normalized pixel value, and x min and x max are the minimum and maximum values ​​of the data respectively. After the above steps, the CT image, PET image and Label image are all preprocessed to a size of 144px×144px×144px.

[0099]

[0100] Step S2: construct a parallel co-attention (PCA) module;

[0101] The PCA module constructed in this embodiment is used to deeply fuse the input CT image and PET image, aiming to enhance the joint feature expression of multimodal images through the attention mechanism and improve the segmentation effect of tumors and other pathological areas in medical images. The PCA module captures the correlation features between CT and PET through joint learning of the two, and enhances the model's comprehensive perception of different modal information. It specifically includes the following steps:

[0102] Step S21, inputting the CT image features and the PET image features into the SE module respectively, and outputting the feature weighted vectors of the CT image and the PET image respectively;

[0103] Specifically, in this embodiment, the input CT image feature F ct and PET image feature F pet , whose shapes are Where B is the batch size, C is the number of channels, D, H, and W are depth, height, and width, respectively. The feature weighted vectors Q and V are obtained by performing channel-level feature enhancement through the SE module (Squeeze-and-Excitation Module).

[0104] The SE module obtains global features through global average pooling Then, after a fully connected layer with a ReLU activation function, we get Then the weight se is generated through the sigmoid activation function, and finally se is multiplied element by element with the input feature as the output of the entire module:

[0105]

[0106]

[0107] Among them, F ct and F pet are CT image features and PET image features respectively; X is the global feature; GAP is the global average pooling; T is the feature vector output by the fully connected layer; FC is the fully connected layer; σ1 is the ReLU activation function, σ2 is the sigmoid activation function; se is the weight; ⊙ is the element-by-element multiplication operation; Q and V are the feature weighted vectors of CT image features and PET image features respectively.

[0108] Step S22, calculating an affinity matrix according to the feature weighted vectors of the CT image and the PET image;

[0109] Affinity=tanh(W b (Q)·W b (V)) (4)

[0110] Among them, Affinity is the affinity matrix; tanh is the activation function; W b The affinity matrix of this embodiment is used to quantify the similarity between the CT image and the PET image, and helps the model capture the mutual correlation between the two.

[0111] Step S23: Calculate the co-attention weights of the CT image features and the PET image features according to the affinity matrix;

[0112] H v =tanh(W v (V)+W q (Q) Affinity) (5)

[0113] H q =tanh(W q (Q)+W v (V) Affinity) (6)

[0114] a v =softmax(W h ·H v ) (7)

[0115] a q =softmax(W h ·H q ) (8)

[0116] Among them, H v , H q are the modality enhancement feature vectors of PET image and CT image respectively, where and W v , W q , W h are weight parameters respectively; a v and a q are the co-attention weights of PET image features and CT image features, respectively. and

[0117] Step S24: according to the common attention weight a v and a qThe weighted sum of the PET and CT features is calculated respectively. After the features of the two modalities are weighted, we get and As the output feature. ct and F′ pet Splice along the channel number dimension to obtain the fused common attention feature map

[0118] F′ ct =∑a q ·Q (9)

[0119] F′ pet =∑a v ·V (10)

[0120] F fusion =Concat(F′ ct ,F′ pet ) (11)

[0121] The entire PCA module converts two shapes into The modal features of are fused together, and the final output combines the attention weighted feature map of the two modal information, with a shape of

[0122] Step S3, inserting several parallel co-attention mechanism modules into the backbone network of the UNet3D network model to construct an improved UNet3D network model;

[0123] After constructing the PCA module, this embodiment further constructs an improved UNet3D network model. The core idea of ​​this model is to extract features by inputting CT and PET image data into two independent UNet3D encoders respectively, and fuse the CT and PET features using a parallel co-attention module (PCA) at an appropriate location. The fused features are then input into the backbone network for further processing, and the final output is the segmentation task result of the dual-modal image. This network model enhances the interaction between different modal features by appropriately inserting the PCA module into the encoder part of the backbone UNet3D, thereby improving the feature expression ability and model performance, which specifically includes the following sub-steps:

[0124] Step S31, inputting the CT image and the PET image into two independent UNet encoder stages, respectively, outputting preliminary features q1, q2, q3, and q4 of the CT image, and outputting preliminary features v1, v2, v3, and v4 of the PET image;

[0125] In this embodiment, each encoder stage includes multiple 3D convolutional layers of 3×3×3 size, BatchNormalization, PReLU activation function and Dropout operation, and the number of output channels of each encoding stage is 16, 32, 64, and 128 respectively.

[0126] Step S32, inserting a parallel co-attention mechanism module into the back end of each encoder stage of the backbone network of UNet3D;

[0127] In this embodiment, the PCA module is applied to the back end of each encoder stage of the backbone network for the preliminary features of CT and PET images. The PCA module enhances the network's ability to characterize features of different modalities by establishing the relationship between the features of the two modalities. Specifically, the PCA module includes the following contents:

[0128] The original CT image and PET image data are input into the first parallel co-attention mechanism module, and the fused bimodal feature F0 is output;

[0129] The preliminary features q1 of the CT image and v1 of the PET image obtained in the first stage are input into the second parallel co-attention mechanism module, and the fused bimodal features F1 are output;

[0130] The preliminary features q2 of the CT image and v2 of the PET image obtained in the second stage are input into the third parallel co-attention mechanism module, and the fused bimodal features F2 are output;

[0131] The preliminary features q3 of the CT image and v3 of the PET image obtained in the third stage are input into the fourth parallel co-attention mechanism module, and the fused bimodal features F3 are output;

[0132] The preliminary features q4 of the CT image and v4 of the PET image obtained in the fourth stage are input into the fifth parallel co-attention mechanism module, and the fused bimodal features F4 are output;

[0133] Step S33, splicing the dual-modal feature F0 and the original CT image and PET image data to form a feature vector with 4 channels, and inputting it into the encoder module of the first stage. After feature extraction by the encoder module, the number of channels becomes 16;

[0134] The concatenated feature map output by the encoder module in the first stage is concatenated with F1 and input to the encoder module in the second stage. After feature extraction by the encoder module, the number of channels becomes 32;

[0135] The concatenated feature map output by the second stage encoder module is concatenated with F2 and input into the third stage encoder module. After feature extraction by the encoder module, the number of channels becomes 64;

[0136] The concatenated feature map output by the encoder module in the third stage is concatenated with F3 and input into the encoder module in the fourth stage. After feature extraction by the encoder module, the number of channels becomes 128;

[0137] The concatenated feature map output by the fourth stage encoder module is concatenated with F4 and input into the fifth stage encoder module. After feature extraction by the encoder module, the number of channels becomes 256.

[0138] Step S34: In the decoder part, the output of the encoder module in the fifth stage is deconvolved to restore the number of channels from 256 to 128, and the output is concatenated with the feature map of the encoder module in the fourth stage, and features are extracted through convolution operation;

[0139] The output of the decoder module in the fourth stage is deconvolved to restore the number of channels from 128 to 64, and the output is concatenated with the feature map of the encoder module in the third stage, and features are extracted through convolution operations;

[0140] The output of the decoder module in the third stage is deconvolved to restore the number of channels from 64 to 32, and the output is concatenated with the feature map of the encoder module in the second stage, and features are extracted through convolution operations;

[0141] The output of the decoder module of the second stage is deconvolved to restore the number of channels from 32 to 16, and the output is concatenated with the feature map of the encoder module of the first stage, and features are extracted through convolution operation;

[0142] The output of the decoder module in the first stage is passed through a 1x1x1 convolutional layer to reduce the number of channels from 16 to 2, and the segmentation task result is output.

[0143] At this point, the network model architecture has been constructed. Through this network structure design, the CT and PET image features are fully integrated and interacted at each encoder stage. The introduction of the PCA module enhances the feature complementarity between CT and PET, which helps to improve the network's representation ability and segmentation performance for dual-modality images. In the backbone network, through layer-by-layer feature extraction, fusion, decoding and recovery, the final efficient segmentation result is achieved.

[0144] Step S4: using the combined loss function to optimize the improved UNet3D network model, which specifically includes the following contents:

[0145] In order to optimize the training process of the network, this embodiment adopts a combined loss function, including the Soft Dice Loss loss function and the weighted Focal Loss loss function. Among them, Soft Dice Loss can effectively deal with the problem of class imbalance in the data, and optimize the performance of the model by measuring the overlap between the predicted results and the true labels; while Focal Loss further enhances the focus on difficult-to-classify samples by introducing α and γ parameters, and reduces the impact of easy-to-classify samples on gradient updates. The specific combined loss function is defined as follows:

[0146] Loss=Soft Dice Loss+0.5×Focal Loss (12)

[0147] Among them, Focal Loss is defined as:

[0148] FocalLoss(p)=-α(1-p) γ log(p) (13)

[0149] Among them, α=0.5 and γ=3. The selection of these two parameters can adjust the weights of difficult-to-classify samples and easy-to-classify samples, making the network more stable when dealing with category imbalance.

[0150] The present invention adopts the Adam optimizer and introduces the cosine annealing learning rate scheduling strategy in the training process. This strategy can dynamically adjust the learning rate to avoid falling into the local optimal solution. Specifically, the learning rate shows a periodic decay with the increase of training rounds, and the initial learning rate is 1×10 -4 , the minimum learning rate is set to 1×10 -9 The model training is divided into multiple rounds. In each round, the prediction value is calculated through forward propagation, and the automatic mixed precision (AMP) technology is combined to reduce the memory usage and improve the training efficiency.

[0151] Step S5: Input the preprocessed head and neck tumor dataset into the optimized improved UNet3D network model, update the network weight parameters, and output the segmentation task results of the dual-modal image.

[0152] Example 2

[0153] This embodiment is based on the method steps of Example 1, and takes the head and neck tumor (HEAD and NECK TumOR, HECKTOR) dataset released by MICCAI in 2022 as an example to clearly and completely describe the technical solution in the embodiment of the present invention, which specifically includes the following contents:

[0154] Dataset introduction: The HECKTOR 2022 dataset is a medical image segmentation dataset focusing on head and neck PET / CT data, containing 524 case training data from 7 different data centers. Each case includes CT images, PET images, and real label images of tumors and lymph nodes annotated by professional doctors, and saved in NIFTI format. The original information of the dataset images is shown in Table 1, where the grayscale value of the PET image is the average value of each data center.

[0155] Table 1 Dataset original information

[0156]

[0157] Step S1: Reference Figure 1 , the obtained HECKTOR 2022 dataset is preprocessed, including resampling, image cropping, and image normalization:

[0158] Step S11, resampling;

[0159] The spatial voxel spacing of the input CT image, PET image, and Label image is resampled to 1mm×1mm×1mm, and the spatial coordinates of the CT image are used as the unified benchmark of the three images to ensure that the voxel spacing of the three images is consistent and the spatial coordinate systems are aligned, so that they can be accurately matched in space. The specific operation steps are as follows: First, for CT images and PET images, the trilinear interpolation method is used for resampling to maintain the smoothness and continuity of image details, and to ensure that their grayscale information and subtle structures are not lost during the resampling process. For Label images, since they contain discrete category values, in order to avoid confusion of category values, the nearest neighbor interpolation method is used for resampling to ensure that the category values ​​remain unchanged after resampling and to avoid the introduction of non-integer intermediate values ​​in the interpolation process. In the resampling process, the spatial direction, starting origin, and coordinate system of the CT image are uniformly used, and the size of the resampled image is adjusted accordingly according to its original voxel spacing to ensure that the spatial resolution of the three images is consistent after resampling and they can be strictly aligned in the same spatial coordinate system.

[0160] Step S12, image cropping;

[0161] The resampled Label image is cropped. First, the pixels (lymph nodes) with a label value of 2 are relabeled as background values ​​0, and only the background (value 0) and the tumor (value 1) are retained. Then, the spatial dimensions of the CT image, PET image, and Label image are cropped to reduce the image size, reduce the computational pressure of subsequent model training, and retain the effective information area containing the tumor. The specific operation steps include: first, by detecting the spatial position of the tumor in the Label image, determine the bounding box (Bounding Box) where the tumor is located. According to the position of the bounding box, the center point of the tumor is calculated, and then the CT image, PET image, and Label image are cropped to a fixed size of 144px×144px×144px based on the center point to ensure that the tumor area is completely retained. At the same time, the image boundary is taken into account during the cropping process to ensure that the cropping operation does not exceed the actual size of the image. If the cropped area is smaller than the expected size, the starting point is adjusted appropriately to ensure that the final cropped image and label meet the predetermined size requirements.

[0162] Step S13, image normalization;

[0163] The PET image is z-score normalized, the mean and standard deviation of the PET image are calculated, and each pixel is normalized according to formula (1).

[0164] Next, the grayscale value (Hounsfield units, HU) of the CT image is truncated to limit its value to the range of [-1024, 1024]. This can avoid the influence of extreme values ​​in the CT image on subsequent processing and retain the common medical image data range. Finally, the maximum-minimum normalization (Min-Max Normalization) is used to normalize the CT image and map its grayscale value to [-1, 1] to ensure the consistency of the image data range. The maximum-minimum normalization formula can refer to formula (2). After the above steps, the CT image, PET image and Label image are all preprocessed to a size of 144px×144px×144px.

[0165] After the above operations, the case information provided by each data center is processed to a uniform voxel spacing, uniform shape size, and uniform grayscale threshold. Figure 2 This is a comparison chart of a case in the data set before and after preprocessing. Table 2 is the data set information after preprocessing, where the grayscale value of PET is the overall mean.

[0166] Table 2 Dataset information after preprocessing

[0167]

[0168]

[0169] Step S2: Construct a parallel co-attention (PCA) module, refer to Figure 3 , the algorithm steps of the parallel co-attention mechanism module are as follows:

[0170] Step S21, input image feature processing;

[0171] For the input CT image feature F ct and PET image feature F pet , whose shapes are Where B is the batch size, C is the number of channels, D, H, and W are depth, height, and width, respectively. The feature weighted vectors Q and V are obtained by performing channel-level feature enhancement through the SE module (Squeeze-and-Excitation Module). The structure diagram of the SE module is shown in Figure 4 As shown, the calculation formula can refer to formula (3), and the global feature is obtained by global average pooling Then, after a fully connected layer with a ReLU activation function, we get Then the weight se is generated through the sigmoid activation function, and finally se is multiplied element by element with the input feature as the output of the entire module.

[0172] Step S22, affinity matrix calculation;

[0173] The affinity matrix is ​​calculated using formula (4) for the Q and V features obtained after processing by the SE module: This affinity matrix is ​​used to quantify the similarity between CT and PET, helping the model capture the correlation between the two.

[0174] Step S23, calculation of co-attention features: further interactively operate the CT and PET features to calculate their co-attention features. Combined with the affinity matrix Affinity, we calculate them respectively through formulas (5)-(8): and Then calculate the attention weights under the two modes and Normalization is performed within each modality using the softmax function.

[0175] Step S24, attention weighted feature generation: Finally, according to the common attention weight a v and a q Calculate the weighted sum of PET and CT features respectively. After weighting the features of the two modalities, refer to formulas (9)-(11) to obtain and As the output feature. ct and F′ pet Splice along the channel number dimension to obtain the fused common attention feature map

[0176] The entire PCA module converts two shapes into The modal features of are fused together, and the final output combines the attention weighted feature map of the two modal information, with a shape of

[0177] Step S3: Build an improved network model and improve the framework structure, refer to Figure 5 , which specifically includes the following contents:

[0178] Step S31: Figure 6 As shown in the figure, CT and PET image data are input into two independent UNet encoder stages respectively. The CT data is processed by the feature extraction module of each stage to obtain the preliminary features q1 to q4 of the CT image. The PET data is also processed by an independent feature extraction module to obtain preliminary features v1 to v4. These features will be fused in the subsequent PCA module. Each encoder stage contains multiple 3D convolutional layers of size 3×3×3, Batch Normalization, PReLU activation function and Dropout operation. The number of output channels of each encoding stage is 16, 32, 64, and 128 respectively.

[0179] Step S32, for the preliminary features of CT and PET images, apply PCA modules at the back end of each encoder stage of the backbone network. The first PCA module directly processes the original CT and PET data and outputs the fused bimodal feature F0. Subsequently, after the feature extraction of the CT and PET images in the first stage, the obtained features (q1 and v1) are respectively input into the second PCA module to obtain the bimodal feature F1. Similarly, after the feature extraction of the second, third and fourth stages, CT and PET are respectively input into the third, fourth and fifth PCA modules to obtain F2, F3 and F4, respectively, to achieve cross-modal fusion of CT and PET features at each stage. The PCA module enhances the network's ability to characterize features of different modalities by establishing the relationship between the features of the two modalities.

[0180] Step S33, in the backbone UNet3D network, firstly, the fused CT and PET features F0 and their original data are spliced ​​to form a feature vector with 4 channels and input into the encoder module of the first stage. After feature extraction, the number of channels becomes 16, and then spliced ​​with F1 and input into the next encoder stage. Next, the spliced ​​features are input into the second stage encoder, and the number of feature channels is further increased to 32. Similarly, the feature map processed by the second stage encoder is spliced ​​with F2 and input into the third stage encoder, and the number of channels is increased to 64. This process is carried out in each stage in turn to ensure that the cross-modal fusion features of the PCA module can be combined in each stage, and the number of feature channels increases to 256 at the end of the encoder stage.

[0181] Step S34, after the encoder stage processing of the backbone UNet3D network is completed, enter the decoder stage. In the decoder part, the spatial resolution of the feature map is first gradually restored through deconvolution operations. Specifically, the last layer output of the encoder first restores the number of channels from 256 to 128 through deconvolution, and then splices with the features from the fourth stage encoder, and further extracts features through convolution operations. Subsequently, similar operations are repeated in each stage of the decoder, gradually restoring the resolution of the feature map, and splicing the output of each stage with the feature map of the corresponding stage of the encoder. Finally, after processing by the decoder, the number of channels is reduced to 2 through a 1x1x1 convolution layer, and then after the sigmoid activation function, the output result is the final predicted segmentation map.

[0182] Step S4: Input the preprocessed data into the above-constructed network model for processing to obtain the predicted segmentation result, verify the training effect and evaluate the performance of the model, and save the optimal trained network model.

[0183] In this invention, in order to evaluate the effectiveness of the proposed PCA-UNet model, a comparative experiment was designed and conducted. The experiment used UNet3D as the baseline network, and compared and analyzed the inputs of CT single modality, PET single modality, and PET-CT dual modality, respectively. Multiple evaluation criteria were used, including DSC (Dice Similarity Coefficient), HD (Hausdorff Distance), Precision, and Recall, to comprehensively measure the segmentation performance of the model. At the same time, an experiment was designed to introduce the PCA module before the input layer of the baseline network to explore the improvement of the model effect by the PCA module.

[0184] Throughout the experiment, the following indicators were used to evaluate the performance of the network:

[0185] The Dice similarity coefficient (DSC) is used to measure the overlap between the predicted segmentation area and the true label area. The calculation formula is:

[0186]

[0187] Among them, P represents the predicted segmentation area, and T represents the actual label area. The higher the DSC value, the closer the segmentation result is to the actual tumor area.

[0188] Hausdorff distance (HD), used to evaluate the maximum distance difference between the predicted segmentation boundary and the true boundary, is defined as:

[0189] HD(P,T)=max{sup p∈P inf t∈T ∥pt∥,sup t∈T inf p∈P ∥tp∥} (15)

[0190] Among them, P represents the point set of the predicted segmentation area, that is, the boundary point set of the tumor area generated by the model, T represents the point set of the real segmentation area, that is, the boundary point set of the real tumor area, ∥pt∥ represents the Euclidean distance between two points p and t, sup represents the maximum distance in the point set (supremum), and inf represents the minimum distance value in the point set (infimum).

[0191] Precision is used to measure the accuracy of the model in positive class prediction and is defined as:

[0192]

[0193] Among them, TP is a true positive example and FP is a false positive example. The higher the precision, the better the segmentation accuracy of the model.

[0194] Recall is used to evaluate the coverage of the model on positive samples. The calculation formula is:

[0195]

[0196] Among them, TP is a true positive example and FN is a false negative example. The higher the recall rate, the more tumor areas the model can capture.

[0197] Implementation details of this experiment: The experiment is based on Python 3.8, Pytorch 2.3.0 and cuda 11.8, using a 16GB NVIDIA 4060TI GPU, batch_size is set to 3, the number of training epochs is set to 300, the Adam optimizer is used, and the cosine annealing learning rate scheduling strategy is introduced during the training process. This strategy can dynamically adjust the learning rate to avoid falling into the local optimal solution. Specifically, the learning rate shows periodic decay with the increase of training rounds, and the initial learning rate is 1×10 -4 , the minimum learning rate is set to 1×10 -9 The model training is divided into multiple rounds. In each round, the prediction value is calculated through forward propagation, and the automatic mixed precision (AMP) technology is combined to reduce the memory usage and improve the training efficiency.

[0198] In order to optimize the network training process, this example uses the SoftDiceLoss+0.5×Focal Loss combined loss function, including Soft Dice Loss and weighted Focal Loss. Among them, Soft Dice Loss can effectively deal with the problem of class imbalance in data, and optimize the performance of the model by measuring the overlap between the predicted results and the true labels; while Focal Loss further enhances the focus on difficult-to-classify samples by introducing α and γ parameters, reducing the impact of easy-to-classify samples on gradient updates.

[0199] The training and evaluation of all models are performed under the same settings to ensure the fairness and repeatability of the comparative experiments. The results of the comparative experiments are shown in the following table.

[0200] Table 3 Experimental results

[0201]

[0202] In the table, PCA-UNet is the model constructed by the present invention, and PCA+UNet3D is a model that introduces a PCA module before the input layer of the baseline network. The PCA-UNet model proposed by the present invention is superior to the baseline model UNet3D and the improved model PCA+UNet3D in all indicators. Specifically, PCA-UNet improves the DSC indicator by about 3% compared with the best result of the baseline UNet3D, showing a significant improvement in segmentation accuracy; in terms of the HD indicator, PCA-UNet shows a smaller error distance, and the optimization range is about 7%, showing its improvement in boundary accuracy. For Precision and Recall, PCA-UNet improved by about 1% and 4% respectively, further enhancing the model's ability to recognize and recall tumor areas. For the overall parameter quantity increased from 5648386 of the baseline network to 6650262 of PCA-UNet, the overall parameter quantity only increased by about 1 million. The overall results show that PCA-UNet can better integrate multimodal information and effectively improve the segmentation performance and boundary clarity of the model, fully verifying its advantages in head and neck tumor segmentation.

[0203] Step S5: Input the pre-processed CT images and PET images of the patient's head and neck tumors into the trained optimal network model, update the network weight parameters, and obtain the optimal network segmentation result.

[0204] like Figure 7 The figure shows the comparison of tumor segmentation using the optimal model trained with each model. The segmentation comparison results show that the PCA-UNet model performs better in detail restoration and contour fitting in tumor segmentation. Specifically, PCA-UNet generates more detailed and smooth edges in segmentation, and the segmented area is closer to the actual tumor contour, effectively capturing small edge features. In comparison, the baseline network has obvious detail loss under single-modal input, especially the single-modal results of PET and CT, and cannot accurately depict the tumor edge. In this way, PCA-UNet is not only more adaptable to multi-modal information, but also can greatly improve the precision and accuracy of segmentation, and better serve the needs of precise segmentation in practical applications.

[0205] Although the specific implementation of the invention is described in detail in conjunction with the drawings, it should not be understood as limiting the scope of protection of this patent. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of this patent.

Claims

1. A head and neck tumor image segmentation method based on CT and PET joint analysis, characterized in that: The following steps are involved: S1, obtaining a head and neck tumor dataset including CT images, PET images and segmented label images, and preprocessing the head and neck tumor dataset; S2. Build a parallel co-attention mechanism module; S3, inserting several parallel co-attention mechanism modules into the backbone network of the UNet3D network model to construct an improved UNet3D network model; S4, using the combined loss function to optimize the improved UNet3D network model; S5. Input the preprocessed head and neck tumor dataset into the optimized improved UNet3D network model, and output the segmentation task results of the dual-modal image.

2. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 1, characterized in that: In S1, the neck tumor dataset is preprocessed, including the following steps: S11, resampling the CT image and the PET image using a triple linear interpolation method, and resampling the label image using a nearest neighbor interpolation method; when sampling the CT image, the PET image, and the label image, adjusting the size of the resampled image based on the spatial direction, the starting origin, and the coordinate system of the CT image, until the voxel spacing of the CT image, the PET image, and the label image is consistent and the spatial coordinate systems are aligned; S12, detecting the spatial position of the tumor in the label image, determining the bounding box where the tumor is located, calculating the center point of the tumor according to the position of the bounding box, and cropping the CT image, PET image, and label image to a size of 144px×144px×144px based on the center point; S13, performing z-score normalization processing on the PET image, and performing grayscale value truncation and maximum-minimum normalization processing on the CT image in sequence.

3. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 1, characterized in that: The S2 comprises the following sub-steps: S21, inputting the CT image features and the PET image features into the SE module respectively, and outputting the feature weighted vectors of the CT image and the PET image respectively; S22, calculating an affinity matrix according to the feature weighted vectors of the CT image and the PET image; S23, calculating the co-attention weights of the CT image features and the PET image features respectively according to the affinity matrix; S24. Calculate the weighted sum of the CT image features and the PET image features according to their co-attention weights, respectively, and concatenate the weighted sum of the CT image features and the PET image features to obtain a fused co-attention feature map.

4. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 3, characterized in that: In S21, the feature weighted vectors of the CT image and the PET image are output, including: Among them, F ct and F pet are CT image features and PET image features respectively; X is the global feature; GAP is the global average pooling; T is the feature vector output by the fully connected layer; FC is the fully connected layer; σ1 is the ReLU activation function, σ2 is the sigmoid activation function; se is the weight; ⊙ is the element-by-element multiplication operation; Q and V are the feature weighted vectors of CT image features and PET image features respectively.

5. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 4, characterized in that: In S22, an affinity matrix is ​​calculated, including: Affinity=tanh(W b (Q)·W b (V)) Among them, Affinity is the affinity matrix; tanh is the activation function; W b is the weight matrix.

6. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 5, characterized in that: In S23, the co-attention weights of the CT image features and the PET image features are calculated respectively, including: H v =tanh(W v (V)+W q (Q)·Affinity) H q =tanh(W q (Q)+W v (V)·Affinity) a v =softmax(W h ·H v ) a q =softmax(W h ·H q ) Among them, H v , H q are the modality enhancement feature vectors of PET image and CT image respectively; W v , W q , W h are weight parameters respectively; a v and a q are the co-attention weights of PET image features and CT image features, respectively.

7. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 6, characterized in that: In S24, the fused common attention feature map is obtained, including: F′ ct =∑a q ·Q F′ pet =∑a v ·V F fusion =Concat(F′ ct ,F′ pet ) Among them, F′ ct and F′ pet is the output feature after weighting CT image features and PET image features; Concat is the concatenation operation; F fusion It is the fused co-attention feature map.

8. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 1, characterized in that: The S3 comprises the following sub-steps: S31, inputting the CT image and the PET image into two independent UNet encoder stages, respectively, outputting preliminary features q1, q2, q3, and q4 of the CT image, and outputting preliminary features v1, v2, v3, and v4 of the PET image; S32, insert a parallel co-attention mechanism module into the back end of each encoder stage of the backbone network of UNet3D; The original CT image and PET image data are input into the first parallel co-attention mechanism module, and the fused bimodal feature F0 is output; The preliminary features q1 of the CT image and v1 of the PET image obtained in the first stage are input into the second parallel co-attention mechanism module, and the fused bimodal features F1 are output; The preliminary features q2 of the CT image and v2 of the PET image obtained in the second stage are input into the third parallel co-attention mechanism module, and the fused bimodal features F2 are output; The preliminary features q3 of the CT image and v3 of the PET image obtained in the third stage are input into the fourth parallel co-attention mechanism module, and the fused bimodal features F3 are output; The preliminary features q4 of the CT image and v4 of the PET image obtained in the fourth stage are input into the fifth parallel co-attention mechanism module, and the fused bimodal features F4 are output; S33, splicing the dual-modal feature F0 and the original CT image and PET image data to form a feature vector with 4 channels, and inputting it into the encoder module of the first stage, and the number of channels becomes 16 after feature extraction by the encoder module; The concatenated feature map output by the encoder module in the first stage is concatenated with F1 and input to the encoder module in the second stage. After feature extraction by the encoder module, the number of channels becomes 32; The concatenated feature map output by the second stage encoder module is concatenated with F2 and input into the third stage encoder module. After feature extraction by the encoder module, the number of channels becomes 64; The concatenated feature map output by the encoder module in the third stage is concatenated with F3 and input into the encoder module in the fourth stage. After feature extraction by the encoder module, the number of channels becomes 128; The concatenated feature map output by the fourth stage encoder module is concatenated with F4 and input into the fifth stage encoder module. After feature extraction by the encoder module, the number of channels becomes 256. S34, in the decoder part, the output of the encoder module of the fifth stage is deconvolved to restore the number of channels from 256 to 128, and the output is concatenated with the feature map of the encoder module of the fourth stage, and features are extracted through convolution operation; The output of the decoder module in the fourth stage is deconvolved to restore the number of channels from 128 to 64, and the output is concatenated with the feature map of the encoder module in the third stage, and features are extracted through convolution operations; The output of the decoder module in the third stage is deconvolved to restore the number of channels from 64 to 32, and the output is concatenated with the feature map of the encoder module in the second stage, and features are extracted through convolution operations; The output of the decoder module of the second stage is deconvolved to restore the number of channels from 32 to 16, and the output is concatenated with the feature map of the encoder module of the first stage, and features are extracted through convolution operation; The output of the decoder module in the first stage is passed through a 1x1x1 convolutional layer to reduce the number of channels from 16 to 2, and the segmentation task result is output.

9. The head and neck tumor image segmentation method based on CT and PET joint analysis according to claim 1, characterized in that: In S4, the combined loss function includes a Soft Dice Loss loss function and a weighted Focal Loss loss function, which is specifically: Loss=Soft Dice Loss+0.5×Focal Loss Focal Loss(p)=-α(1-p) γ log(p) Among them, Loss is the combined loss function; p is the confidence of the model prediction; α and γ are parameters.

Images