Deep learning based fully automatic segmentation and benign / malignant classification method for small renal masses

Abstract

The application relates to a deep learning-based full-automatic segmentation and benign-malignant classification method for kidney small tumors, which comprises the following steps: S1, dividing a 3D kidney image into left and right part images and flipping; S2, a segmentation model is used for segmenting the original image and the flipped image to obtain a predicted segmentation image; the predicted segmentation images are weightedly averaged to obtain a fused tumor predicted segmentation image, the fused tumor predicted segmentation images are compared, a fused tumor predicted segmentation image with a larger tumor prediction area is identified as containing a tumor, and the tumor segmentation result is placed in the original image and the flipped image corresponding to the side to form a segmentation image to be classified; S3, the segmentation image to be classified is input into a classification model to predict the tumor class in each segmentation image to be classified, and the final benign-malignant classification result is obtained by averaging and fusing the tumor classes in the segmentation images to be classified. The application can realize efficient and accurate segmentation and classification of kidney tumors.

Description

Technical Field

[0001] This invention relates to magnetic resonance kidney image recognition and processing technology in the medical field, and particularly to a method and apparatus for fully automated segmentation and benign / malignant classification of small kidney tumors based on deep learning. Background Technology

[0002] With the widespread adoption and development of imaging technology, the detection rate of small renal mass (SRM) has significantly improved, accounting for approximately 66% of all kidney tumors. However, previous studies have shown that about 20-30% of surgically resected SRMs are benign tumors, indicating that preoperative benign or malignant diagnosis of SRMs still faces significant challenges. Furthermore, the learning curve for young physicians to become experienced radiologists is lengthy, and the increasing workload of imaging diagnosis in recent years has become a problem that cannot be ignored in clinical practice.

[0003] Thanks to its high soft tissue resolution, multi-parameter, and multi-sequence imaging capabilities, MRI can better assess the internal structure and invasion of tumors, demonstrating higher efficacy than CT in the diagnosis and differential diagnosis of renal cell carcinoma (RCC). However, current research on MRI-based artificial intelligence models for the diagnosis of benign and malignant SRM is limited or limited to SRM segmentation, and there is still no reliable method to achieve automatic SRM segmentation and benign / malignant classification based on MRI images. Therefore, developing a fully automated SRM segmentation and classification system based on MRI has significant clinical implications. Summary of the Invention

[0004] To address the problems in the existing technology, this application proposes a fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning, which includes steps S1-S3:

[0005] S1: Divide the current 3D kidney image containing the tumor of the patient into left and right parts according to the depth direction (Z axis), and flip the two parts of the image in multiple directions. Then input the original images of the left and right parts and the flipped images into the segmentation model; the tumor is a solitary tumor.

[0006] S2: The segmentation model segments the original and flipped images of the input left and right parts of the image based on image recognition, and obtains tumor prediction segmentation maps of the left and right parts of the image in different directions; the tumor prediction segmentation maps are weighted and averaged to obtain fused tumor prediction segmentation maps corresponding to the left and right parts of the image; the fused tumor prediction segmentation maps of the left and right parts of the image are compared, and the fused tumor prediction segmentation map of the side with the larger tumor prediction region is identified as containing a tumor, and the tumor segmentation result is placed in the original and flipped images corresponding to that side to form the segmentation maps to be classified; the segmentation model is based on the U-Net neural network architecture, its feature extraction layer is based on the EfficientNet_b4 model, and a combination of Robust Cross EntropyLoss and Soft Dice Loss is used as the loss function;

[0007] S3: Input the segmentation map to be classified into the classification model. The classification model predicts the tumor category in each segmentation map to be classified, and obtains the final benign / malignant classification result by averaging and fusing the tumor categories in each segmentation map to be classified. The classification model structure includes an EfficientNet_B4 backbone extraction network, a 4-layer fully connected network, a three-layer ReLU function, and two dropout layers with a dropout_prob of 0.5.

[0008] According to some embodiments of the present invention, dividing a current 3D kidney image containing a tumor of a patient into left and right parts along the depth direction (Z-axis) includes: receiving a current 3D kidney image containing a tumor of the patient, preprocessing the image, the preprocessing including: automatically calculating an Otsu threshold, obtaining a kidney binary mask based on the Otsu threshold, and cropping the image based on the kidney binary mask to obtain left and right parts of the image.

[0009] According to some embodiments of the present invention, the automatic calculation of the Otsu threshold includes steps S111-S114:

[0010]

[0011]

[0012] According to some embodiments of the present invention, the preprocessing further includes employing min-max normalization to map the voxel values ​​of the input kidney image to the range of [0, 1] to unify the contrast and brightness of the image and eliminate scale differences between different images, as shown in the following formula:

[0013]

[0014] According to some embodiments of the present invention, the current 3D kidney images containing tumors in the patient include T2WI, T1WI, CP, NP and DP images.

[0015] According to some embodiments of the present invention, the segmentation model consists of an encoder, a decoder, and skip connections; the input of the segmentation model is a convolutional layer used to convert an image with 1 channel into a feature map with 48 channels, which is input to the encoder; the encoder consists of the first 22 layers of EfficientNet_B4; the decoder consists of consecutive convolutional layers and transposed convolutional layers; each layer of the decoder is connected to the corresponding layer of the encoder through skip connections; the feature map output by the decoder is finally input into a convolutional layer to output the segmentation map to be classified.

[0016] According to some embodiments of the present invention, in addition to the convolutional layer at the output end, a ReLU activation function and a normalization layer are added after each newly added convolutional layer and transposed convolutional layer; and a dropout layer with dropout_prob of 0.5 is added after the first and second ReLU layers.

[0017] According to some embodiments of the present invention, the classification model uses EfficientNet_B4 pre-trained weights during training and uses the spatial cross-entropy function as the loss function; the weight decay rate is 0.0005 and the initial learning rate is 0.001; the gradient descent algorithm is performed using the Adam optimizer with momentum and adaptive learning rate.

[0018] This application also proposes a deep learning-based fully automated segmentation and benign / malignant classification device for small renal tumors, which includes a processor and a memory; the processor executes the fully automated segmentation and benign / malignant classification method for small renal tumors based on a computer program stored in the memory.

[0019] This application also proposes a computer program product, including a computer program that, when executed by a processor, implements the steps of the described fully automated segmentation and benign / malignant classification method for small renal tumors.

[0020] Through the above embodiments of the present invention, efficient and accurate segmentation and classification of kidney tumors can be achieved while taking into account both model performance and computational power consumption. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used are briefly described below:

[0022] Figure 1 A flowchart illustrating a fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning, according to some embodiments of the present invention, is shown.

[0023] Figure 2 A flowchart of renal tumor data preprocessing according to some embodiments of the present invention is shown;

[0024] Figure 3 This diagram illustrates a segmentation model structure according to some embodiments of the present invention.

[0025] Figure 4 A schematic diagram comparing manual segmentation and automatic segmentation achieved by the present invention is shown.

[0026] Figure 5 The diagram shows the DICE similarity coefficient density of the segmentation model according to some embodiments of the present invention on an external test set;

[0027] Figure 6 A structural diagram of a classification model according to some embodiments of the present invention is shown;

[0028] Figure 7 The ROC curves of the classification model are shown on different datasets;

[0029] Figure 8 The ROC curves of an automatic segmentation classification model and a manually segmented classification model based on some embodiments of the present invention are shown on an external test set below 3 cm.

[0030] Figure 9 A schematic structural diagram of a deep learning-based fully automated segmentation and benign / malignant classification device for small kidney tumors is shown, according to some embodiments of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] Before introducing and explaining the technical solution of this application, the relevant terms involved in the embodiments of this application will be introduced and explained.

[0033] Batch Normalization Layer (BN layer): A technique that provides zero-mean / unit-variance input to any layer in a neural network, used to improve the performance and stability of artificial neural networks.

[0034] Dropout layer: During the forward propagation of a neural network, some neurons are made to have an activation value of 0 with a certain probability to prevent the model from overfitting.

[0035] Segmentation model: A mathematical model obtained by training labeled samples using machine learning techniques. During training, the parameters in the model are updated according to the data samples. When performing a segmentation task, the parameters of the mathematical model are loaded and the segmentation result of the input data is calculated and generated.

[0036] Classification model: A mathematical model obtained by training labeled samples using machine learning techniques. During training, the parameters in the model are updated according to the data samples. When performing a classification task, the parameters of the mathematical model are loaded and the classification result of the input data is calculated and generated.

[0037] Feature maps: intermediate results in deep learning neural networks after being processed by specific modules (such as convolutional layers).

[0038] ReLU function: A non-linear activation function in deep learning that returns 0 when the input value is less than 0 and returns the input value when it is greater than 0.

[0039] Gray-level histogram: A function of gray levels, describing the number of pixels at each gray level in an image, reflecting the frequency of each gray level in the image.

[0040] Between-class variance is a metric used in statistics and pattern recognition to measure the degree of difference between different categories. It assesses the separability of categories by calculating the difference between the means of different categories and the population mean.

[0041] Encoder: Transforms input data into a low-dimensional, dense latent representation, that is, extracts the core features of the data.

[0042] Parameter sharing: The feature extraction layer of the classification model shares a set of model parameters.

[0043] Decoding layer: Reconstructs the latent representation output by the encoding layer into the original data or generates new data.

[0044] Feature information fusion: Feature information fusion refers to the effective combination of features from different sources, different levels or different modalities.

[0045] Robust Cross Entropy Loss is an improvement on traditional cross-entropy loss, designed to enhance the robustness of models in complex scenarios such as noisy data, class imbalance, or outlier samples.

[0046] Soft Dice Loss: Soft Dice loss is a commonly used loss function in segmentation tasks, used to directly optimize the overlap between model predictions and the true labels.

[0047] DICE similarity coefficient: used to evaluate the similarity between two sets, with a value range of [0, 1]. The higher the value, the better the overlap.

[0048] Modality shift refers to a situation in multimodal learning or cross-modal tasks where there are significant differences in the data distribution or feature representation of different modalities, making it difficult for the model to effectively align or fuse information from these modalities.

[0049] Adversarial mapping refers to the learning of bidirectional mappings between different modalities, domains, or data distributions through adversarial training techniques. It aims to eliminate distributional differences between the source and target domains or achieve cross-modal feature alignment.

[0050] ROC curve: This refers to a curve that describes how the classification performance of a binary classification model changes as the decision threshold changes, with the false positive rate on the horizontal axis and the true positive rate on the vertical axis.

[0051] Area under the curve (AUC): Used to evaluate the diagnostic performance of a classification model. The value ranges from [0, 1]. The higher the value, the better the classification performance of the model.

[0052] Figure 1 A flowchart illustrating a fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning, according to some embodiments of the present invention, is shown. As shown in the figure, the method includes the following steps:

[0053] S1: Divide the current 3D kidney image containing the tumor of the patient into left and right parts according to the depth direction (Z axis), and flip the two parts of the image in multiple directions. Input the original image and the flipped image of the left and right parts into the segmentation model; the tumor is a solitary tumor.

[0054] The 3D kidney images here refer to the raw images from five magnetic resonance imaging (MRI) sequences: T2WI, T1WI, CP, NP, and DP images.

[0055] The left and right images refer to the images of the left and right kidneys. The original image is the image before flipping, and the flipped image includes horizontally flipped, vertically flipped, and horizontally and vertically flipped images. A solitary tumor refers to a tumor that doctors have determined in the early stages, through various methods, to exist only on the left or right side (the specific side is unknown) rather than both sides.

[0056] According to some embodiments of the present invention, dividing a current 3D kidney image containing a tumor of a patient into left and right parts along the depth direction (Z-axis) includes: receiving a current 3D kidney image containing a tumor of the patient, preprocessing the image, the preprocessing including: automatically calculating an Otsu threshold, obtaining a kidney binary mask based on the Otsu threshold, and cropping the image based on the kidney binary mask to obtain left and right parts of the image.

[0057] The minimum bounding box (bbox) of the kidney image is extracted based on the binarized image, and the binarized image (i.e., Figure 2 The kidney binarization mask shown is used to obtain a cropped image, i.e., to obtain a 3D kidney image containing the tumor and the kidney region. This reduces background interference and provides high-quality, accurate input data for subsequent tumor segmentation and data processing, improving the segmentation accuracy and efficiency of kidney tumor segmentation tasks. The automatic calculation of the Otsu threshold (i.e., the optimal threshold) includes:

[0058]

[0059]

[0060] According to some embodiments of the present invention, the preprocessing may further include employing min-max normalization (e.g., ... Figure 2 As shown, the voxel values ​​of the input kidney image are mapped to the range [0, 1] to unify the image contrast and brightness, eliminate scale differences between different images, enhance the effect of the Otsu thresholding method, improve the accuracy of image binarization, and provide consistent input data for subsequent model training, thus promoting model stability and convergence speed. The formula is as follows:

[0061]

[0062] S2: The segmentation model performs steps S21-S23 on the original image and the flipped image in the left and right parts;

[0063] S21: The segmentation model segments the original and flipped images of the left and right parts of the input image based on image recognition, and obtains tumor prediction segmentation maps of the left and right parts of the image in different directions.

[0064] The different orientations include images in their original positions, as well as images after horizontal flipping, vertical flipping, and horizontal-vertical flipping.

[0065] S22: Perform weighted average fusion on the tumor prediction segmentation map to obtain the fused tumor prediction segmentation map corresponding to the left and right parts of the image. Compare the fused tumor prediction segmentation map of the left and right parts of the image. The fused tumor prediction segmentation map of the side with the larger tumor prediction area is identified as containing the tumor. The tumor segmentation result is placed in the original image and the flipped image corresponding to that side to form the segmentation map to be classified.

[0066] In this step, if the tumor prediction region corresponding to the fusion result of the left image is larger than the tumor prediction region corresponding to the fusion result of the right image, the tumor is considered to be in the left image; otherwise, the tumor is considered to be in the right image. Finally, the segmentation result of the side where the tumor is located is used as the final predicted segmentation result and placed in the original image and the flipped image of that side (for example, the tumor segmentation line is highlighted in the original image). The image thus formed is used as the segmentation map for subsequent tumor classification.

[0067] The method described above reduces the segmentation model's over-reliance on certain features (which are insufficient when processing only the original image), thus enhancing the model's generalization ability. Furthermore, by segmenting left and right and retaining only the image on the side containing the tumor for subsequent processing, the amount of images requiring final processing is significantly reduced. Moreover, this invention eliminates false positives during tumor segmentation. Specifically, the segmentation model itself may segment tumors in areas where no tumor exists. Since this invention targets solitary kidney tumors, by comparing the sizes of the two segmented left and right sides, tumors mistakenly segmented by the model can be cleverly excluded.

[0068] The segmentation model also uses the above methods during training. In addition, other rotation or transformation methods besides those mentioned above are used to enhance the data during training, which will not be elaborated here.

[0069] The segmentation model is based on the U-Net neural network architecture, with its feature extraction layer based on the EfficientNet_b4 model, and uses a combination of Robust Cross Entropy Loss and Soft Dice Loss as the loss function. The inventors found that this design balances model performance and computational cost. Robust Cross Entropy Loss focuses on classifying each pixel of the kidney image, while Soft Dice Loss focuses on the overlap between the segmented kidney region and the ground truth region. Combining the two further ensures the accuracy of the segmentation results.

[0070] like Figure 3As shown, the model consists of an encoder, a decoder, and skip connections. First, the model's input is a convolutional layer that converts a 1-channel image into a 48-channel feature map. This feature map is then fed into the encoder for further extraction of higher-order features. The encoder consists of the first 22 layers of EfficientNet_B4, progressively extracting features and reducing the size of the feature map. Pre-trained model parameters are directly loaded during the initial training phase to efficiently extract features from the input image. Subsequently, the features extracted by the encoder are fed into the decoder to restore the feature map resolution. The decoder consists of successive convolutional layers and upsampling layers (deconvolutional or transposed convolutional layers) that progressively enlarge the feature map size until it reaches the spatial dimensions of the input image. Each convolutional layer consists of multiple convolutional kernels to capture features at different scales in the kidney image. Each layer of the decoder is connected to the corresponding layer of the encoder via skip connections to maintain consistency of contextual information. Skip connections fuse feature information from different levels by connecting encoders and decoders, enabling the model to utilize both low-level and high-level features simultaneously, thereby improving segmentation accuracy and preserving local details. The feature map output from the decoder is ultimately fed into a convolutional layer to output the final segmentation map. To prevent overfitting, in addition to the output convolutional layer, this application adds a ReLU activation function and a normalization layer after each newly added convolutional layer and transposed convolutional layer.

[0071] The training method for this segmentation model is described in further detail below.

[0072] This application uses EfficientNet_B4 pre-trained weights to train the tumor segmentation network. A combination of RobustCross EntropyLoss and Soft Dice Loss is used as the loss function. As mentioned earlier, Robust CrossEntropyLoss focuses on classifying each pixel of the kidney image, while Soft Dice Loss focuses on the overlap between the segmented kidney region and the ground truth region. Combining the two further ensures the accuracy of the segmentation results.

[0073] To avoid poor local minima and achieve better local minima, this invention chooses an initial learning rate of 0.0001. The gradient descent algorithm is executed using the Adam (Adaptive Moment Estimation) optimizer with momentum and an adaptive learning rate. The network model is trained for 100 epochs, with batch training (batch size 20) performed within each epoch. The loss is calculated, and the model parameters are updated via backpropagation. Validation and evaluation are performed after each epoch, calculating the DICE similarity coefficient, and adjusting the learning rate based on the results. If the current model performs better, the best model is saved, and this process is repeated every 10 epochs.

[0074] Figure 4 The diagram illustrates a comparison between manual segmentation and automatic segmentation achieved by the present invention, where the cases are three cases from different centers. Manual segmentation is indicated in green, automatic segmentation in red, and the overlapping portion is indicated in blue. Figure 5 The diagram shows the DICE similarity coefficient density map of the segmentation model of the present invention on the external test set. The median (Q1, Q3) of DICE for the five sequences in the external test set are 0.813 (0.730, 0.880), 0.729 (0.493, 0.797), 0.806 (0.696, 0.847), 0.821 (0.728, 0.865), and 0.814 (0.744, 0.868), respectively. The median of DICE for the five sequences in the internal test set are 0.822 (0.720, 0.867), 0.714 (0.388, 0.813), 0.828 (0.744, 0.867), 0.836 (0.714, 0.877), and 0.824 (0.738, 0.871).

[0075] S3. Input the segmentation map to be classified into the classification model. The classification model predicts the tumor category in each segmentation map to be classified, and obtains the final benign or malignant classification result by averaging and fusing the tumor categories in each segmentation map to be classified. The classification model structure includes an EfficientNet_B4 backbone extraction network, a 4-layer fully connected network, a three-layer ReLU function, and two dropout layers with a dropout_prob of 0.5.

[0076] The inference strategy of this application is Test-Time Augmentation. As mentioned above, the segmentation model segments the original and flipped images of the input left and right parts of the image based on image recognition, obtains tumor prediction segmentation maps of the left and right parts of the image in different directions, performs weighted average fusion of the tumor prediction segmentation maps to obtain fused tumor prediction segmentation maps corresponding to the left and right parts of the image, compares the fused tumor prediction segmentation maps of the left and right parts of the image, and identifies the fused tumor prediction segmentation map of the side with the larger tumor prediction area as containing a tumor, and puts the tumor segmentation result into the original and flipped images of the corresponding side to form each segmentation map to be classified; accordingly, the classification model uses each segmentation map to be classified in different directions (original, horizontally flipped, vertically flipped, horizontally and vertically flipped) to predict the tumor category in each segmentation map, and obtains the final benign or malignant classification result by averaging the tumor categories in the classification image.

[0077] To address the heterogeneity of MRI scanners across different hospitals and devices, according to some embodiments of the present invention, the segmented images to be classified undergo preprocessing. This preprocessing includes image alignment using a CycleGAN model to reduce modal shifts caused by differences in scanners and scanning parameters. This model, by learning the generative adversarial mapping relationship between source and target modalities, can convert MRI images from different sources into a unified spatial voxel distribution while maintaining a high degree of consistency in their anatomical structures and tissue features. This model effectively achieves cross-device image alignment, ensuring spatial consistency of the converted images while mitigating modal inconsistencies caused by device differences.

[0078] The classification model structure of this application includes an EfficientNet_B4 backbone extraction network, a 4-layer fully connected network, three ReLU function layers, and two dropout layers with a dropout_prob of 0.5. Through experiments with models from EfficientNet_B0 to EfficientNet_B7, the inventors selected the EfficientNet_B4 model, which balances model efficiency and performance. Previous studies have used EfficientNet_B4 for tumor classification, but no studies have applied it to multi-sequence MRI examination of kidney tumors. EfficientNet_B4 can balance the model's depth, width, and resolution through a compound scaling method. This compound scaling method helps the automatic classification model extract richer image features from kidney images, while also improving model accuracy and reducing the number of parameters. First, the number of input channels in the original first convolutional layer of the EfficientNet_B4 model is changed to 1. To adapt to the classification categories of this application, the original classification layer of the model is discarded, and the feature maps extracted by the model are input into a completely new classification layer, which includes the classification of benign and malignant tumors. In the EfficientNet_B4 backbone extraction network, this application employs a parameter contribution strategy to reduce model complexity. T2WI, T1WI, CP, NP, and DP images with 1 input channel are transformed into feature maps with 1792 channels after passing through the EfficientNet_B4 backbone extraction network. The feature maps of each modality are concatenated according to their channel count and input into the fully connected layer, changing the feature dimensions from (1, 1792×5) to (1, 1792), (1, 512), (1, 256), and (1, 2). To prevent overfitting, this application adds a dropout layer with dropout_prob of 0.5 after the first and second ReLU layers (e.g., ...). Figure 6 (As shown). Figure 6 The parameter sharing shown means that in this classification model, all five MRI sequences are used to extract features through the same classification network to achieve parameter sharing. That is, features are extracted from the five MRI sequences and fused to complete the binary classification prediction of benign and malignant.

[0079] The following describes the training method for the classification model. This application uses EfficientNet_B4 pre-trained weights and spatial cross-entropy as the loss function during training. The weight decay rate is 0.0005. To avoid poor local minima and achieve better local minima, this embodiment chooses an initial learning rate of 0.001. The gradient descent algorithm is executed using the Adam (Adaptive MomentEstimation) optimizer with momentum and an adaptive learning rate. The classification model is trained for 100 epochs, with batch training (batch size 32) within each epoch. The loss value is calculated for each epoch, and the model parameters (such as weights, biases, etc.) are updated through backpropagation to reduce the loss. Model evaluation is performed at each epoch during training to calculate the classification accuracy. If the current model performs better, the best model is saved.

[0080] Figure 7 The ROC curves of the classification model on different datasets are shown. Figure 8 The ROC curves of the classification model based on automatic segmentation and the classification model based on manual segmentation are shown on the external test set below 3 cm. The model has an AUC of 0.936 (0.889 - 0.967) on the internal test set and an AUC of 0.786 (95% CI: 0.678 - 0.871) on the external test set, indicating that the classification model of this invention has high sensitivity.

[0081] Figure 9 A schematic structural diagram of a deep learning-based fully automated segmentation and benign / malignant classification device 500 for small renal tumors, representing some embodiments of the present invention, is shown below. Figure 9 As shown, the device includes a processor 51, a memory 52, and a bus 53.

[0082] In some instances, the device may further include an input device 501, an input port 502, an output port 503, and an output device 504. The input port 502, processor 51, memory 52, and output port 503 are interconnected via a bus 53. The input device 501 and output device 504 are connected to the bus 53 via input port 502 and output port 503, respectively, and thus connected to other components of the device. It should be noted that the output and input interfaces here can also be represented by I / O interfaces. Specifically, the input device 501 receives input information from an external source, such as an image, and transmits the input information to the processor 51 via the input port 502. The processor 51 processes the input information based on computer-executable instructions stored in the memory 52 to generate output information, temporarily or permanently storing the output information in the memory 52, and then transmitting the output information to the output device 504 via the output port 503. The output device 504 outputs the output information to the outside of the device. The output information may include benign or malignant tumor classification results.

[0083] The aforementioned memory 52 includes a large-capacity memory for data or instructions.

[0084] Bus 53 includes hardware, software, or both, which couples multiple components together.

[0085] The processor 51 executes a deep learning-based fully automated segmentation and benign / malignant classification method for small kidney tumors based on the computer program stored in the memory 52.

[0086] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the deep learning-based fully automated segmentation and benign / malignant classification method for small renal tumors.

[0087] The detailed descriptions of the figures mentioned above are included here by reference and will not be repeated here.

[0088] Through the above embodiments of the present invention, efficient and accurate segmentation and classification of kidney tumors can be achieved while taking into account both model performance and computational power consumption.

[0089] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning, characterized in that, Including steps S1-S3: S1: Divide the current 3D kidney image containing the tumor of the patient into left and right parts according to the depth direction, and flip the two parts of the image in multiple directions. Then input the original images of the left and right parts and the flipped images into the segmentation model; the tumor is a solitary tumor. S2: The segmentation model segments the original and flipped images of the input left and right parts of the image based on image recognition, and obtains tumor prediction segmentation maps of the left and right parts of the image in different directions; the tumor prediction segmentation maps are weighted and averaged to obtain fused tumor prediction segmentation maps corresponding to the left and right parts of the image; the fused tumor prediction segmentation maps of the left and right parts of the image are compared, and the fused tumor prediction segmentation map of the side with the larger tumor prediction region is identified as containing a tumor, and the tumor segmentation result is placed in the original and flipped images corresponding to that side to form the segmentation maps to be classified; the segmentation model is based on the U-Net neural network architecture, its feature extraction layer is based on the EfficientNet_b4 model, and a combination of Robust Cross EntropyLoss and Soft Dice Loss is used as the loss function; S3: Input the segmentation map to be classified into the classification model. The classification model predicts the tumor category in each segmentation map to be classified, and obtains the final benign / malignant classification result by averaging and fusing the tumor categories in each segmentation map to be classified. The classification model structure includes an EfficientNet_B4 backbone extraction network, a 4-layer fully connected network, a three-layer ReLU function, and two dropout layers with a dropout_prob of 0.

5.

2. The method for fully automated segmentation and benign / malignant classification of small renal tumors based on deep learning according to claim 1, characterized in that, Dividing a current 3D kidney image containing a tumor from a patient into left and right parts along the depth direction includes: receiving the current 3D kidney image containing a tumor from the patient, preprocessing the image, the preprocessing including: automatically calculating the Otsu threshold, obtaining a kidney binary mask based on the Otsu threshold, and cropping the image based on the kidney binary mask to obtain the left and right parts of the image.

3. The fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning according to claim 2, characterized in that, The automatic calculation of the Otsu threshold includes steps S111-S114:

4. The fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning according to claim 2, characterized in that, The preprocessing also includes using min-max normalization to map the voxel values ​​of the input kidney image to the range of [0, 1], in order to unify the contrast and brightness of the image and eliminate scale differences between different images, as shown in the following formula: , The maximum value representing a voxel. The minimum value representing a voxel.

5. The fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning according to claim 1, characterized in that, The patient's current 3D kidney images containing tumors include T2WI, T1WI, CP, NP, and DP images.

6. The fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning according to claim 1, characterized in that, The segmentation model consists of an encoder, a decoder, and skip connections. The input of the segmentation model is a convolutional layer, which is used to convert an image with 1 channel into a feature map with 48 channels. This feature map is then input to the encoder. The encoder consists of the first 22 layers of EfficientNet_B4. The decoder consists of consecutive convolutional layers and transposed convolutional layers. Each layer of the decoder is connected to the corresponding layer of the encoder through skip connections. The feature map output by the decoder is eventually fed into a convolutional layer to output the segmentation map to be classified.

7. The fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning according to claim 6, characterized in that, In addition to the convolutional layers at the output, a ReLU activation function and a normalization layer are added after each newly added convolutional layer and transposed convolutional layer; and a dropout layer with dropout_prob of 0.5 is added after the first and second ReLU layers.

8. The fully automated segmentation and benign / malignant classification method for small renal tumors based on deep learning according to claim 1, characterized in that, The classification model uses EfficientNet_B4 pre-trained weights during training and uses the spatial cross-entropy function as the loss function; the weight decay rate is 0.0005 and the initial learning rate is 0.001; the gradient descent algorithm is performed using the Adam optimizer with momentum and adaptive learning rate.

9. A fully automated segmentation and benign / malignant classification device for small renal tumors based on deep learning, characterized in that, It includes a processor and a memory; the processor executes the deep learning-based fully automated segmentation and benign / malignant classification method for small renal tumors as described in any one of claims 1-8, based on a computer program stored in the memory.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the deep learning-based fully automated segmentation and benign / malignant classification method for small renal tumors as described in any one of claims 1-8.

Images