A neural network model for lymphoma PET image 3D segmentation task

Abstract

The application discloses a neural network model for a lymphoma PET image 3D segmentation task, which is based on a five-layer "U"-type encoder-decoder structure, the encoder part of each layer is fused with a cyclic residual convolution module to deepen the convolution depth and enhance the feature extraction capability, and a multi-scale convolution module is used to enhance the feature of the feature image output by the encoder before feature fusion. The layers are connected through a maximum pooling layer and an up-sampling layer, the maximum pooling layer can compress the features and simplify the network, and the up-sampling layer can retain the high-level abstract features while improving the image resolution, and then the high-resolution image of the low-level surface features enhanced by the multi-scale convolution module is used for feature fusion.

Description

Technical Field

[0001] This invention relates to a neural network model for 3D segmentation of lymphoma PET images. Background Technology

[0002] Lymphoma is a malignant tumor originating from lymphocytes in the lymphohematopoietic system. According to statistics from the World Health Organization, the incidence of lymphoma has been rapidly increasing over the past two decades, with an annual growth rate of approximately 5%. Currently, lymphoma has become one of the ten most common malignant tumors in China. It can be mainly divided into two categories: non-Hodgkin lymphoma (NHL) and Hodgkin lymphoma (HL). The incidence of non-Hodgkin lymphoma is significantly higher than that of Hodgkin lymphoma, accounting for approximately 90% of all lymphoma cases. Non-Hodgkin lymphomas all originate from B-cell lesions and can be further subdivided into diffuse large B-cell lymphoma, follicular lymphoma, and mantle cell lymphoma. Diffuse large B-cell lymphoma is an aggressive lymphoma, accounting for 31%-34% of non-Hodgkin lymphomas, and even exceeding 40% in Asian countries. It not only has a high relapse rate but also has a refractory rate in 50% of patients, resulting in a very severe prognosis. The results of the SCHOLAR-1 study showed that the clinical remission rate for patients with relapsed, refractory diffuse large B-cell lymphoma was only 26%, and the median survival was only 6.3 months.

[0003] The lymphatic system is distributed throughout the body, and lymphoma symptoms vary depending on the location of the lymphoma. Furthermore, lymphoma subtypes are extremely complex, with over 80 different subtypes. This leads to long diagnosis times and high misdiagnosis rates, severely impacting patient treatment. Different lymphoma subtypes exhibit significant differences in treatment intensity and prognosis, making accurate diagnosis crucial in clinical practice. Traditionally, the most accurate diagnosis of lymphoma relied on pathological examination, where doctors examined tissue samples under a microscope – a time-consuming process dependent on the doctor's experience and condition. With advancements in medical imaging technology, techniques such as positron emission tomography (PET), computed tomography (CT), and magnetic resonance imaging (MRI) are widely used in medical diagnosis. Among these, PET has become one of the primary methods for diagnosing and locating lymphoma.

[0004] In PET images, tumor cells in the human body exhibit far greater metabolic activity than normal cells, absorbing large amounts of labeled metabolites. The concentration of the labeled substance can be determined by the standardized uptake value (SUV), which is the brightness level in the PET image, thus distinguishing lesion areas. However, due to the low resolution and grayscale nature of PET images, the randomness of lymphoma size, shape, location, and number, and interference from normal human tissues, precise segmentation of lymphoma areas by doctors becomes increasingly difficult. Traditional methods of manually locating and segmenting lymphomas not only consume a significant amount of doctors' time and energy but also involve many subjective factors. The doctor's experience and condition can affect the segmentation results, leading to misdiagnosis or missed diagnosis.

[0005] Whether through pathological examination or PET image observation, the diagnosis of lymphoma relies heavily on the doctor's experience and condition. Reducing misdiagnosis and missed diagnosis caused by subjective factors is key to improving the accuracy of lymphoma diagnosis. In recent years, with the rapid development of deep learning-based segmentation algorithms, their high degree of automation and significant segmentation results have created new possibilities for fully automated medical image segmentation and diagnosis.

[0006] In the FCN model proposed by Long et al., convolution is used instead of fully connected layers, and deconvolution is used to map the feature map to the original image size, achieving end-to-end fully automatic segmentation for the first time. Badrinayanan et al. proposed the SegNet model based on FCN, proposing a symmetric network structure of encoder-decoder and using dilated convolution to expand the field of view and improve segmentation accuracy. Ronneberger et al. proposed the u-net network with encoder-decoder structure for medical image segmentation, achieving significant segmentation results on biomedical image sets. Cicek et al., building upon the u-net network, replaced 2D operations with 3D operations, proposing the 3DUnet network to achieve 3D segmentation of the u-net model. Segmentation tests were conducted on the African Xenopus kidney dataset, achieving an average IoU (Intersection over Union) of 0.863 in a 3x crossover experiment. Milletari et al. proposed the VNet network, which not only generalized the u-net model from 2D to 3D but also incorporated residual modules and used Dice Loss to efficiently complete prostate MRI segmentation tasks. Zhou et al., based on a depth-supervised encoder-decoder network, proposed the UNet++ network, redesigning lateral connection paths and integrating features from different levels to improve accuracy. Isensee et al. proposed the nnU-net network, which automatically adapts to any dataset, achieving superior segmentation of the u-net architecture. Experiments show that nnU-net can adaptively and efficiently complete any new 3D segmentation task without manual parameter tuning, demonstrating strong generalization capabilities.

[0007] Among the methods described above, the encoder-decoder u-net architecture achieves high segmentation accuracy in biomedical image segmentation tasks with small data volumes and simple semantics. Furthermore, 3DUnet implements 3D segmentation using the u-net architecture, VNet integrates a residual module into a 3D u-net, and nnU-net can adaptively and efficiently complete any new 3D segmentation task without manual parameter tuning. However, the lymphoma PET images studied in this paper exhibit randomness in lymphoma size, shape, location, and number, resulting in the lymphoma 3D segmentation task not achieving ideal segmentation accuracy in existing 3DUnet, VNet, and nnU-net models. While UNet++ integrates features from different levels to some extent, it suffers from high model complexity and is unsuitable for the characteristics of lymphoma PET. Summary of the Invention

[0008] The main objective of this invention is to provide a neural network model for 3D segmentation of lymphoma PET images, so as to improve the segmentation accuracy of lymphoma with limited GPU memory.

[0009] To achieve the above objectives, this invention provides a neural network model for 3D segmentation of lymphoma PET images. The neural network model is based on a U-shaped encoder-decoder structure with five layers. Each layer's encoding portion includes a recurrent residual convolutional module and a multi-scale convolutional module. The recurrent residual convolutional module encodes the input image, and the multi-scale convolutional module enhances the features of the encoder output. Adjacent encoder layers are connected by a max-pooling layer and an upsampling layer. The max-pooling layer compresses features and simplifies the network, while the upsampling layer improves the resolution of high-level abstract feature images and performs data augmentation, facilitating feature fusion with the high-resolution low-level surface features enhanced by the multi-scale convolutional module. Each layer also includes a decoder portion for decoding the fused encoded image.

[0010] As a further improvement of the present invention, the cyclic residual convolution module includes a cyclic residual convolution layer and a 3×3×3 convolution layer. The cyclic residual convolution layer is composed of a 3×3×3 convolution layer combined with the residual module cyclic convolution twice.

[0011] As a further improvement of the present invention, the multi-scale convolution module has four layers. The first layer consists of a 1×1×1 convolutional layer, a 1×1×1 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 1. The second layer consists of a 1×1×1 convolutional layer, a 3×3×3 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 2. The third layer consists of a 1×1×1 convolutional layer, a 5×5×5 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 4. The fourth layer consists of a 1×1×1 convolutional layer, a 7×7×7 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 8.

[0012] As a further improvement of the present invention, the multi-scale convolution module, starting from the second layer, combines the output of the previous layer with the output of the 1×1×1 convolutional layer of the current layer, and then inputs it into the subsequent convolutional layers.

[0013] As a further improvement of the present invention, the output of the cyclic residual convolution module is directly input into the multi-scale convolution module, which has a total of 4 layers and the number of layers can be cropped according to the size of the input image.

[0014] As a further improvement of the present invention, each 1×1×1 convolutional layer of the multi-scale convolutional module reduces the number of image input channels N to N / T, where T is the number of layers selected by the multi-scale convolutional module; finally, the outputs of all layers are fused together to restore the number of channels N, and then output through a single 1×1×1 convolutional layer.

[0015] As a further improvement of the present invention, the neural network model can also combine the features output by the multi-scale convolution module with the low-level decoding features and input them into the decoder for decoding. The decoder consists of two 3×3×3 convolutional layers.

[0016] As a further improvement of the present invention, the neural network model also incorporates FocalLoss and DiceLoss as loss functions, with the specific mathematical expression as follows:

[0017]

[0018] In the Loss expression, a and b are the weight constants of FocalLoss and DiceLoss, and N is the number of data points for each training iteration; in the FocalLoss expression, y∈{1,-1} is the ground truth category, p∈[0,1] is the model's prediction probability for y=1, and γ≥0 is an adjustable focusing parameter; in the DiceLoss expression, sets X and Y are the predicted values ​​or the true values, respectively.

[0019] As a further improvement of the present invention, the learning rate adjustment strategy of the neural network model uses a power function, the specific mathematical expression of which is:

[0020]

[0021] Where lr is the initial learning rate, n is the current number of training iterations, maxEpoch is the maximum number of training iterations, and γ is the exponent.

[0022] The beneficial effects of this invention are as follows: The neural network model for 3D segmentation of lymphoma PET images of this invention combines the u-net architecture, the recurrent residual convolution module, and the multi-scale convolution module. This allows the recurrent residual convolution module to increase the convolution depth and enhance the feature extraction capability. Furthermore, the multi-scale convolution module is used to enhance the feature image output by the encoder before feature fusion. In this way, the segmentation accuracy of lymphoma is improved under limited GPU memory. Attached Figure Description

[0023] Figure 1 This is a structural diagram of the neural network model of this invention.

[0024] Figure 2 This is a structural diagram of the cyclic residual convolution module of the present invention.

[0025] Figure 3 This is a structural diagram of the multi-scale convolution module of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] It should be emphasized that, in describing this invention, various formulas and constraints are distinguished by consistent reference numerals, but it is not excluded that different reference numerals may be used to identify the same formulas and / or constraints. The purpose of this arrangement is to more clearly illustrate the features of this invention.

[0028] Because of the random location of lymphomas, the entire human PET image needs to be trained for segmentation. The large amount of data for a single case means that even though the 3DUnet model uses a lightweight u-net architecture, it still places high demands on device memory for lymphoma 3D segmentation tasks.

[0029] In view of this, this invention proposes a novel convolutional neural network (CNN) based on the u-net architecture to address the problems existing in lymphoma PET data. Specifically tailored to the characteristics of lymphoma PET images, it integrates the u-net architecture, recurrent residual convolutional modules, and multi-scale convolutional modules into a network model. This is because: the u-net architecture is highly effective in medical image segmentation tasks; its "U"-shaped structure not only effectively alleviates the limitation of GPU memory on image resolution but also effectively solves the problem of small sample sizes in medical image datasets, which is detrimental to training, through data augmentation; considering that the recurrent residual convolutional module can increase network depth without changing parameters, and the multi-scale convolutional module performs excellently in spoofing detection tasks, this invention combines the recurrent residual convolutional module, the multi-scale convolutional module, and 3DU-net to improve the 3D segmentation accuracy of lymphoma with limited GPU memory.

[0030] like Figures 1 to 3 As shown, the neural network model of this invention is based on a U-shaped encoder-decoder structure, which has five layers. Each layer's encoding part includes a recurrent residual convolution module and a multi-scale convolution module. The recurrent residual convolution module is used to increase the convolution depth and enhance feature extraction capabilities; the multi-scale convolution module is used to enhance the feature image output by the encoder before feature fusion. Adjacent encoder layers are connected by a max-pooling layer and an upsampling layer. The max-pooling layer is used to compress features and simplify the network; the upsampling layer is used to improve image resolution while preserving high-level abstract features, and then feature fusion is performed with the high-resolution image of the low-level surface features enhanced by the multi-scale convolution module. Each layer contains a decoder part, which decodes the encoded image after feature fusion.

[0031] In other words, the neural network model is based on a five-layer U-shaped encoder-decoder structure. Each layer first uses a recurrent residual convolutional module to "encode" the input image, i.e., the feature extraction process. The recurrent convolutional structure effectively improves the feature extraction capability of the encoding module. Pooling layers are used to downsample the low-level feature map output by the encoder as the input to the next layer, which simplifies the network. Furthermore, a multi-scale convolutional module is used to enhance the low-level features output by the encoder, using multi-scale convolutional kernels and dilated convolutions to enhance the contours, edges, and other low-level features of the feature map. At the bottom layer, two convolutional layers are used to decode the feature map output by the encoder and upsample the high-level feature map output by the decoder to ensure that the size of the high-level feature map output by the decoder is consistent with the feature map output by the previous multi-scale convolutional module. At the same time, upsampling increases the randomness of the data to a certain extent, which has a implicit enhancement effect on the data. Finally, the low-level feature map output by the multi-scale convolutional module is combined with the high-level feature image output by the decoder and input into the decoder for decoding, enriching the edge information of lesions in each layer of feature map, thereby improving the segmentation accuracy.

[0032] In the neural network model of this invention, the recurrent residual convolution module includes one recurrent residual convolutional layer and one 3×3×3 convolutional layer. The recurrent residual convolutional layer is composed of a 3×3×3 convolutional layer combined with two recurrent convolutions of the residual module. The recurrent residual convolutional layer can increase network depth and extract better features without increasing network parameters; the 3×3×3 convolutional layer is used for further feature extraction.

[0033] In the neural network model of this invention, the multi-scale convolutional module has four layers. The first layer consists of a 1×1×1 convolutional layer, a 1×1×1 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 1. The second layer consists of a 1×1×1 convolutional layer, a 3×3×3 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 2. The third layer consists of a 1×1×1 convolutional layer, a 5×5×5 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 4. The fourth layer consists of a 1×1×1 convolutional layer, a 7×7×7 convolutional layer, and a dilated convolutional layer with a kernel of 3×3×3 and an expansion rate of 8. The multi-scale convolutional module reduces the number of image input channels N to N / T in each 1×1×1 convolutional layer, where T is the number of layers selected by the multi-scale convolutional module. Finally, the outputs of all layers are fused to restore the number of channels N, and then output through another 1×1×1 convolutional layer. Furthermore, starting from the second layer, the multi-scale convolutional module fuses the output of the previous layer with the output of the current 1×1×1 convolutional layer before inputting it into subsequent convolutional layers. Considering the low resolution and varying lesion sizes of PET images, the multi-scale convolutional module of this invention has four layers; however, the specific number of layers can be adjusted according to the feature map size.

[0034] In the neural network model of this invention, the decoder consists of two 3×3×3 convolutional layers. The input of the decoder fuses high-level features from the output of the deep decoder and low-level features from the output of the multi-scale convolutional module. The high-level feature map is upsampled and then fused with the low-level feature map. The low-level features, such as contours and edges, of the low-level feature map can effectively improve the model's segmentation accuracy. The decoder's output is upsampled and used as the high-level feature map input to the decoder of the next layer.

[0035] Considering that FocalLoss can solve the problem of severe imbalance in the ratio of positive to negative samples, the neural network model of this invention also combines FocalLoss and DiceLoss as the loss function Loss, the specific mathematical expression of which is as follows:

[0036]

[0037] In the Loss expression, a and b are the weights of FocalLoss and DiceLoss, and N is the batch size of training data. In the FocalLoss expression, y∈{1,-1} is the ground truth class, p∈[0,1] is the model's prediction probability for y=1, and γ≥0 is an adjustable focusing parameter. In the DiceLoss expression, sets X and Y are the predicted values ​​or the true values, respectively.

[0038] At the same time, the SGD optimizer is used to obtain the loss function that is minimized, and the weights are updated through the backpropagation algorithm.

[0039] The learning rate adjustment strategy of the neural network model uses a power function, and the specific mathematical expression is as follows:

[0040]

[0041] Where lr is the initial learning rate, n is the current number of training iterations, maxEpoch is the maximum number of training iterations, and γ is the exponent.

[0042] The neural network model proposed in this invention for 3D segmentation of lymphoma PET images, based on the U-shaped encoder-decoder structure of the UET model, innovatively integrates recurrent convolution and multi-scale convolution into the 3DUnet model to address the characteristics of low resolution, variable lesion location, and varying lesion size in lymphoma PET images, thereby significantly improving the segmentation accuracy of lymphoma.

[0043] First, the data was preprocessed and randomly divided into training and test sets according to a set ratio. Then, the model was trained using an early stopping method. The model converged well and avoided overfitting, demonstrating good segmentation performance for lymphoma lesions during both training and testing. After preprocessing the lymphoma data, the data was divided into training and test sets and input into a convolutional neural network for training. Lymphoma segmentation was tested on the test set. Comparative experiments showed that the proposed convolutional neural network achieved a similarity coefficient (Dice) of 83.13%, which is significantly improved compared to existing models such as 3DUnet (79.99%), VNet (77.82%), Unet++ (80.20%), and nnUnet (81.60%). Furthermore, it outperformed the comparative models in five evaluation metrics: Jaccard overlap, sensitivity, positive predictive value (PPV), Hausdorff distance (HD95), and average surface distance (ASD), proving that the proposed model has significant advantages in lymphoma segmentation tasks.

[0044] Further ablation experiments showed that the Dice values ​​of the cyclic residual convolution module and the multi-scale convolution module reached 81.10% and 81.66%, respectively, which are significantly improved compared to 3DUnet's 79.99%. Furthermore, significant improvements were observed in five evaluation metrics: Jaccard overlap, sensitivity, positive predictive value (PPV), Hausdorff distance (HD95), and average surface distance (ASD). Specifically, the HD95 and ASD values ​​of the multi-module convolution module reached 5.7068 and 1.0355, respectively, which are significantly improved compared to 3DUnet's 7.9697 and 1.9123, demonstrating that the multi-scale convolution module significantly improves the segmentation accuracy of lymphoma lesion margins. Therefore, the experiments prove that the neural network model of this invention is truly effective.

[0045] Of course, this invention can also increase segmentation accuracy by increasing the number of convolution kernels and the model depth, provided that the graphics card memory allows, without considering time efficiency and model complexity.

[0046] In summary, the neural network model for 3D segmentation of lymphoma PET images of the present invention combines the 3DU-net architecture, a recurrent residual convolution module, and a multi-scale convolution module. This allows the recurrent residual convolution module to increase the convolution depth and enhance the feature extraction capability, and the multi-scale convolution module to enhance the feature image output by the encoder before feature fusion. Thus, the segmentation accuracy of lymphoma lesions is significantly improved under limited GPU memory.

[0047] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A neural network model for 3D segmentation of lymphoma PET images, characterized in that: The neural network model is based on a U-shaped encoder-decoder structure, which has five layers. The encoding part of each layer includes a recurrent residual convolutional module and a multi-scale convolutional module. The recurrent residual convolutional module is used to encode the input image, and the multi-scale convolutional module is used to enhance the feature image output by the encoder. Adjacent encoder layers are connected by max pooling layers, which are used to compress features and simplify the network. Upsampling layers are used to improve the resolution of high-level abstract feature images and data augmentation, facilitating feature fusion with the high-resolution image of low-level surface features enhanced by the multi-scale convolutional module. Each layer contains a decoder section, which is used to decode the encoded image after feature fusion; The neural network model also incorporates FocalLoss and DiceLoss as loss functions, with the specific mathematical expression as follows: In the Loss expression, a and b are the weight constants of FocalLoss and DiceLoss, and N is the number of data points for each training iteration; in the FocalLoss expression, y∈{1, -1} is the ground truth category, p∈[0,1] is the model's prediction probability for y=1, and γ≥0 is an adjustable focusing parameter; in the DiceLoss expression, sets X and Y are the predicted values ​​or the true values, respectively. The learning rate adjustment strategy of the neural network model uses a power function, and the specific mathematical expression is as follows: Where lr is the initial learning rate, n is the current training iterations, maxEpoch is the maximum number of training iterations, and γ is the exponent; The cyclic residual convolution module includes one cyclic residual convolution layer and one 3×3×3 convolution layer. The cyclic residual convolution layer is composed of one 3×3×3 convolution layer combined with two cyclic convolutions of the residual module. The multi-scale convolution module has four layers. The first layer consists of one 1×1×1 convolution layer, one 1×1×1 convolution layer, and a dilated convolution with a 3×3×3 kernel and a dilation rate of 1. The second layer consists of one 1×1×1 convolution layer, one 3×3×3 convolution layer, and a dilated convolution with a 3×3×3 kernel and a dilation rate of 2. The third layer consists of one 1×1×1 convolution layer, one 5×5×5 convolution layer, and a dilated convolution with a 3×3×3 kernel and a dilation rate of 4. The fourth layer consists of one 1×1×1 convolution layer, one 7×7×7 convolution layer, and a dilated convolution with a 3×3×3 kernel and a dilation rate of 8.

2. The neural network model according to claim 1, characterized in that: Starting from the second layer, the multi-scale convolution module combines the output of the previous layer with the output of the 1×1×1 convolutional layer in the current layer, and then inputs it into the subsequent convolutional layers.

3. The neural network model according to claim 1, characterized in that: The output of the cyclic residual convolution module is directly input into the multi-scale convolution module, which has a total of 4 layers and can be cropped according to the size of the input image.

4. The neural network model according to claim 1, characterized in that: The multi-scale convolution module reduces the number of image input channels N to N / T in each 1×1×1 convolutional layer, where T is the number of layers selected by the multi-scale convolution module. Finally, the outputs of all layers are fused to restore the number of channels N, and then output through a 1×1×1 convolutional layer.

5. The neural network model according to claim 1, characterized in that: The neural network model can also combine the features output by the multi-scale convolution module with the underlying decoding features and input them into the decoder for decoding.

Images