Virtual biopsy method for kidney small tumor

Abstract

The invention provides a kidney small tumor virtual biopsy method, and relates to the technical field of deep learning, and the method comprises the steps: collecting clinical CT image data, and carrying out the image registration; constructing and training a kidney tumor automatic detection and segmentation model; carrying out ROI positioning cutting and quality control; and constructing a small kidney cancer virtual biopsy model based on the multi-view feature fusion convolutional neural network, and predicting the probability values of benign and malignant kidney tumors and malignant degrees. Through the technical means of deep learning, the pathological property of the kidney tumor smaller than 4 cm is judged through preoperative enhanced CT, and more bases are provided for selection of treatment schemes of clinicians.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a virtual biopsy method for small renal tumors based on a multi-view feature fusion convolutional neural network. Background Art

[0002] Small renal masses are cystic or solid renal masses no larger than 4 cm in diameter that enhance on CT, MRI, or contrast-enhanced ultrasound. Due to the increasing availability of abdominal imaging, the incidence of small renal masses is increasing worldwide, particularly in the elderly population over 70 years of age with comorbidities. Small renal masses are highly heterogeneous, ranging from completely benign to highly lethal. This presents a clinical dilemma: some benign and malignant renal masses are difficult to distinguish using current imaging modalities. Approximately 30% of patients with small renal masses who undergo partial or radical nephrectomy ultimately have benign pathological findings. More accurate preoperative diagnosis can avoid some unnecessary surgical procedures. Furthermore, a growing number of treatment options are available for malignant small renal masses. A multicenter study demonstrated that active surveillance for small renal cancers does not affect patient survival compared with immediate surgical treatment. Radiofrequency ablation has also become one of the primary treatment options for stage T1a renal cancer. The most important factor in choosing a treatment option is the nature of the small renal mass. For small, low-grade renal cancers, radiofrequency ablation and active surveillance are viable treatment options. Small, high-grade renal cancers are more amenable to immediate surgery. Furthermore, the malignancy of small renal tumors can also provide additional information for selecting partial or radical nephrectomy, as well as the extent of partial nephrectomy.

[0003] However, current imaging methods present significant difficulties in assessing the benign or malignant nature of small renal tumors, as well as their degree of malignancy. For radiologists, some benign pathological types, such as oncocytomas and fat-poor AML, are difficult to distinguish from malignant tumors. Furthermore, the degree of malignancy of a tumor depends on a range of pathological factors, including pathological type, nuclear grade, and surrounding fat invasion. Radiologists are unable to determine the malignancy of most small renal tumors simply by visually examining the images. With the development of artificial intelligence, new image analysis techniques using deep learning are being used to obtain as much information as possible about the lesion site, thereby obtaining results that approach or reach the level of a tissue biopsy diagnosis. This technology is known as virtual biopsy. Currently, there is no virtual biopsy technology that can predict the benign, indolent, or aggressive nature of small renal tumors. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the purpose of the present invention is to provide a virtual biopsy method for small renal tumors. Through deep learning technology, the present invention uses preoperative enhanced CT to determine the pathological nature of renal tumors smaller than 4 cm, providing more basis for clinicians to choose treatment plans.

[0005] In order to solve the above problems, the technical solution of the present invention is: A virtual biopsy method for a small renal tumor comprises the following steps: Collect clinical CT image data and perform image registration; Build and train a renal tumor automatic detection and segmentation model; Perform ROI positioning, cropping and quality control; A small renal cancer virtual biopsy model was constructed based on multi-view feature fusion convolutional neural network to predict the probability of benign and malignant renal tumors and the degree of malignancy.

[0006] Preferably, the steps of collecting clinical CT image data and performing image registration specifically include: collecting clinical patient CT image data, wherein the patient data inclusion criteria are patients who have undergone partial or radical nephrectomy, have a tumor with a maximum diameter of less than 4 cm, and have renal swelling with preoperative enhanced CT, and the CT image data must include arterial phase, venous phase, and plain phase CT images; for the clinically collected plain phase CT images and venous phase CT images, the plain phase CT images and venous phase CT images are registered to the arterial phase CT images by the affine transformation method of the universal registration toolbox in 3D Slicer, thereby realizing multi-phase CT image alignment.

[0007] Preferably, in the step of constructing and training a renal tumor automatic detection and segmentation model, the segmentation network model is a nnU-Net convolutional network, including an encoder branch with 5-level downsampling of 3D convolution and pooling operations, and a decoder branch with 5-level upsampling operations of 3D deconvolution operations and 3D convolution modules, and the encoder and decoder are connected by a jump connection operation.

[0008] Preferably, the steps of performing ROI positioning, cropping, and quality control specifically include: after detecting a renal mass, determining the axial slice containing the largest tumor area based on the segmentation results, extracting a 14 × 14 cm region of interest (ROI) centered on the tumor; in cases involving multiple masses, manually selecting the most invasive one by a reviewer based on a comprehensive review of the pathology report; and resampling the cropped tumor patch to obtain a spatial resolution of 0.625 × 0.625 mm, resulting in an image block of 224 × 224 pixels.

[0009] Preferably, the step of constructing a small renal cancer virtual biopsy model based on a multi-view feature fusion convolutional neural network to predict the probability values ​​of benign and malignant renal tumors and the degree of malignancy specifically includes: inputting multi-view CT image data into a trained multi-view feature fusion convolutional neural network, that is, inputting the ROI images of the arterial phase, venous phase, and plain scan phase CT images into three encoders corresponding to the multi-view feature fusion convolutional neural network, and mapping them into 512-dimensional single-view features respectively; the single-view features of the three views are stacked and sent to the multi-view feature fusion module, and the fused 512-dimensional features are output through a classification network head composed of two fully connected layers to predict the probability values ​​of benign and malignant renal tumors and the degree of malignancy, thereby realizing virtual biopsy of small renal cancer.

[0010] Preferably, the multi-view feature fusion convolutional neural network includes multiple convolutional encoders, a multi-view feature fusion module based on a cross-attention mechanism, and a classification prediction head based on an MLP structure, wherein the convolutional encoder network adopts a ResNet-18 network structure, and the encoders of the three views are independent of each other; for each cross-sectional ROI view, the input corresponding encoder will be mapped into 512-dimensional single-view features respectively, and the single-view features of the three views are stacked and sent to a cross-attention module containing three fully connected layers for feature fusion. The fused 512-dimensional features are output through a classification prediction head composed of two fully connected layers to predict the probability value of benign or malignant renal tumors and the degree of malignancy.

[0011] Preferably, the predicted pathological type is analyzed and output, and the renal tumor is classified into benign, low-grade malignant, and high-grade malignant according to the pathological type.

[0012] Compared with the existing technology, the present invention uses deep learning technology to use preoperative enhanced CT to judge the pathological properties of renal tumors smaller than 4 cm. It can not only predict the benign or malignant nature of renal tumors smaller than 4 cm, but also predict the degree of malignancy of renal tumors, providing more basis for clinicians to choose treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings: Figure 1 This is a flowchart of the virtual biopsy method for small renal tumors of the present invention; Figure 2 This is a detailed flow chart of the virtual biopsy method for small renal tumors of the present invention; Figure 3 Schematic diagram of the renal tumor segmentation network structure; Figure 4 Schematic diagram of the multi-view feature fusion convolutional neural network structure. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0015] Specifically, the present invention proposes a virtual biopsy method for small renal tumors based on multi-view feature fusion convolutional neural network, such as Figure 1 and Figure 2 As shown, the method includes the following steps: S1: Collect clinical CT image data and perform image registration; Specifically, clinical CT image data was collected from patients undergoing partial or radical nephrectomy, with a tumor less than 4 cm in maximum diameter and preoperative enhanced CT scans of the kidney. The CT image data included arterial, venous, and plain CT images. The collected plain and venous CT images were then registered to the arterial CT image using the affine transformation method in the Universal Registration Toolbox in 3D Slicer, achieving multi-phase CT image alignment.

[0016] S2: Build and train a renal tumor automatic detection and segmentation model; Specifically, arterial phase CT images were collected and pixel-level segmentation and annotation were performed by radiologists to mark the kidneys and renal tumor areas as training data. The annotated images were then used to train the segmentation network model. The segmentation network model was as follows: Figure 3 The nnU-Net convolutional network shown in the figure specifically includes an encoder branch with 5-level downsampling of 3D convolution and pooling operations, and a decoder branch with 5-level upsampling operations including 3D deconvolution operations and 3D convolution modules. The encoder and decoder are connected by jump connection operations. The input of the network is the arterial phase CT image, and the model weights are inherited from the pre-trained parameters in the KiTS public dataset. The trained segmentation model can output the corresponding delineation results of the kidney and renal tumor area based on the input arterial phase CT image, and obtain the corresponding segmentation results of the corresponding plain scan and venous phase CT images after registration through the above-mentioned registration results. In the local GPU server or workstation, deploy as follows Figure 2 The trained nnU-Net segmentation network shown outputs the corresponding kidney and renal tumor area delineation results for the remaining arterial phase CT images.

[0017] S3: Perform ROI positioning, cropping and quality control; Specifically, the quality control and ROI cropping process involves the following: After a renal mass is detected, the axial slice containing the largest tumor area is determined based on the segmentation results. In one embodiment, a 14 × 14 cm region of interest (ROI) is extracted centered on the tumor. In cases involving multiple masses, the most aggressive one is manually selected by reviewers based on a comprehensive review of the pathology reports. During the ROI cropping stage, the cropped tumor patch is resampled to a spatial resolution of 0.625 × 0.625 mm, resulting in an image patch of 224 × 224 pixels.

[0018] S4: A small renal cancer virtual biopsy model was constructed based on multi-view feature fusion convolutional neural network to predict the probability of benign and malignant renal tumors and the degree of malignancy.

[0019] Specifically, deploy the following on the local GPU server: Figure 4 The trained multi-view feature fusion convolutional neural network shown in the figure feeds the ROI images of CT images from the arterial, venous, and plain scan phases into its three corresponding encoders, mapping them into 512-dimensional single-view features. The single-view features from the three perspectives are stacked and fed into the multi-view feature fusion module. The fused 512-dimensional features are fed into a classification network head consisting of two fully connected layers to output the predicted probability of renal tumor malignancy.

[0020] like Figure 3 As shown in the figure, the multi-view feature fusion convolutional neural network includes multiple convolutional encoders, a multi-view feature fusion module based on a cross-attention mechanism, and a classification prediction head based on an MLP structure. The convolutional encoder network adopts the ResNet-18 network structure, and the encoders of the three views are independent of each other. For each cross-sectional ROI view, the input to the corresponding encoder will be mapped into 512-dimensional single-view features. After feature stacking, the single-view features of the three views are sent to the cross-attention module containing three fully connected layers for feature fusion. The fused 512-dimensional features are output by the classification prediction head composed of two fully connected layers to predict the probability values ​​of benign and malignant renal tumors and the degree of malignancy.

[0021] The predicted pathological types were analyzed and output, and renal tumors were classified according to the pathological types into benign (angiomyolipomas, complex renal cysts, oncocytomas, mixed stromal epithelial tumors, metanephric adenomas, and other benign tumors such as hemangiomas), low-grade (T1 stage, low nuclear grade, clear cell carcinoma without necrosis or sarcomatous changes, T1 stage, low nuclear grade, papillary cell carcinoma without sarcomatous changes, chromophobe cell carcinoma, clear papillary renal cell carcinoma, multilocular cystic renal cell carcinoma with low malignant potential, epithelioid angiomyolipomas, and other low-grade tumors such as mucotubular spindle cell carcinoma), and high-grade (T3 / 4 stage, or high nuclear grade, or clear cell carcinoma with necrosis / sarcomatous changes, T3 / 4 stage, or high nuclear grade, or papillary cell carcinoma with sarcomatous changes, renal cell carcinoma with TEF-3 rearrangement, unclassified renal cell carcinoma, and other highly malignant renal cancers such as collecting duct carcinoma).

[0022] In this embodiment, for the trained prediction network model, the probability values ​​of benign and malignant renal tumors and the degree of malignancy can be obtained by directly inputting multi-view CT image data into the prediction network model, thereby realizing virtual biopsy of small renal cancer.

[0023] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A virtual biopsy method for small renal tumors, characterized in that: The method comprises the following steps: Collect clinical CT image data and perform image registration; Build and train a renal tumor automatic detection and segmentation model; Perform ROI positioning, cropping and quality control; A small renal cancer virtual biopsy model was constructed based on multi-view feature fusion convolutional neural network to predict the probability of benign and malignant renal tumors and the degree of malignancy.

2. The method for virtual biopsy of small renal tumors according to claim 1, characterized in that: The steps of collecting clinical CT image data and performing image registration specifically include: collecting clinical patient CT image data, wherein the patient data inclusion criteria are patients who have undergone partial or radical nephrectomy, have a tumor with a maximum diameter of less than 4 cm, and have renal swelling with preoperative enhanced CT, and the CT image data must include arterial phase, venous phase, and plain phase CT images; for the clinically collected plain phase CT images and venous phase CT images, registering the plain phase CT images and venous phase CT images to the arterial phase CT images using the affine transformation method of the universal registration toolbox in 3D Slicer to achieve multi-phase CT image alignment.

3. The method for virtual biopsy of small renal tumors according to claim 1, characterized in that: In the step of constructing and training a renal tumor automatic detection and segmentation model, the segmentation network model is a nnU-Net convolutional network, including an encoder branch with 3D convolution and pooling operations and 5-level downsampling, and a decoder branch with 3D deconvolution operations and 5-level upsampling operations of the 3D convolution module, and the encoder and decoder are connected by a jump connection operation.

4. The method for virtual biopsy of small renal tumors according to claim 1, characterized in that: The steps of performing ROI positioning, cropping, and quality control specifically include: after detecting a renal mass, determining the axial slice containing the largest tumor area based on the segmentation results, extracting a 14 × 14 cm region of interest (ROI) centered on the tumor; in cases involving multiple masses, manually selecting the most invasive one by a reviewer based on a comprehensive review of the pathology report; and resampling the cropped tumor patch to obtain a spatial resolution of 0.625 × 0.625 mm, resulting in an image block of 224 × 224 pixels.

5. The method for virtual biopsy of small renal tumors according to claim 1, characterized in that: The steps of constructing a small renal cancer virtual biopsy model based on a multi-view feature fusion convolutional neural network and predicting the probability values ​​of benign and malignant renal tumors and the degree of malignancy specifically include: inputting multi-view CT image data into a trained multi-view feature fusion convolutional neural network, that is, inputting ROI images of arterial phase, venous phase, and plain scan phase CT images into three encoders corresponding to the multi-view feature fusion convolutional neural network respectively, and mapping them into 512-dimensional single-view features; the single-view features of the three views are stacked and sent to the multi-view feature fusion module; the fused 512-dimensional features are output through a classification network head composed of two fully connected layers to predict the probability values ​​of benign and malignant renal tumors and the degree of malignancy, thereby realizing virtual biopsy of small renal cancer.

6. The method for virtual biopsy of small renal tumors according to claim 5, characterized in that: The multi-view feature fusion convolutional neural network includes multiple convolutional encoders, a multi-view feature fusion module based on a cross-attention mechanism, and a classification prediction head based on an MLP structure. The convolutional encoder networks all adopt a ResNet-18 network structure, and the encoders of the three views are independent of each other. For each cross-sectional ROI view, the input to the corresponding encoder will be mapped into 512-dimensional single-view features respectively. The single-view features of the three views are stacked and sent to a cross-attention module containing three fully connected layers for feature fusion. The fused 512-dimensional features are output by a classification prediction head composed of two fully connected layers to predict the probability values ​​of benign and malignant renal tumors and the degree of malignancy.

7. The method for virtual biopsy of small renal tumors according to claim 5, characterized in that: The analysis outputs the predicted pathological type, and renal tumors are divided into benign, low-grade malignant, and high-grade malignant according to the pathological type.

Images