Three-dimensional transrectal ultrasound-based prediction method for response to neoadjuvant therapy in rectal cancer

Abstract

The application discloses a rectal cancer neoadjuvant therapy response prediction method based on three-dimensional transrectal ultrasound, which comprises the following steps: S1, three-dimensional transrectal ultrasound image volume data preprocessing, so as to obtain sampling data of two-dimensional gray scale transrectal ultrasound images in a scanning range; S2, rectal tumor automatic segmentation based on semi-supervised learning, two-dimensional frame sampling data is segmented to output an image of a tumor region of interest; and S3, pathological complete remission prediction based on a 3D-ResNet network, the tumor region is classified and predicted for tumor efficacy, so that high-accuracy automatic prediction of rectal cancer ultrasound images is realized, and precise treatment decision is facilitated.

Description

Technical Field

[0001] This invention relates to a method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound. Background Technology

[0002] Rectal cancer is one of the leading malignant tumors worldwide, with over 700,000 new cases diagnosed annually, a significant proportion of which are classified as locally advanced rectal cancer (LARC, T3-4, or with lymph node involvement). Current guidelines recommend total mesorectal resection for LARC patients after neoadjuvant therapy. However, due to tumor biological heterogeneity, patient responses to treatment vary considerably. In recent years, the "observation and waiting" strategy has gained increasing attention for patients with the potential to achieve pathological complete response (pCR), as it can spare patients from surgery-related complications, preserve organ function, and result in good 5-year survival rates. Therefore, accurately predicting pCR before neoadjuvant therapy has significant clinical implications.

[0003] Previous artificial intelligence (AI) prediction models based on magnetic resonance imaging (MRI) data have achieved good performance. However, MRI equipment is expensive, has long acquisition times, and limited accessibility. Furthermore, these models heavily rely on manual annotation, leading to time-consuming and subjective manual tumor segmentation, which limits their large-scale clinical application. In addition, while traditional two-dimensional transrectal ultrasound (2D TRUS) offers real-time and cost-effective advantages, it is limited by insufficient spatial features and high operator dependence. Three-dimensional transrectal ultrasound (3D TRUS) offers advantages over 2D TRUS, including a wider field of view, the ability to capture complex spatial patterns, and lower operator dependence. Moreover, its equipment cost is lower than MRI, making it easier to deploy in primary healthcare institutions. However, there is currently a lack of automated segmentation tools and pCR prediction models specifically developed for 3D TRUS images of rectal cancer. Summary of the Invention

[0004] This invention overcomes the shortcomings of the prior art and provides a deep learning framework based on 3D TRUS three-dimensional transrectal ultrasound data to realize automatic segmentation of rectal tumor images and prediction methods after pCR neoadjuvant therapy.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound, characterized by: including...

[0007] S1. Preprocessing of three-dimensional transrectal ultrasound image volume data to obtain sampling data of two-dimensional grayscale rectal ultrasound image frames within the scanning range;

[0008] S2. Automatic segmentation of rectal tumors based on semi-supervised learning, which segments two-dimensional frame sampled data and outputs an image outlining the region of interest of the tumor;

[0009] S3. Pathological complete remission prediction based on 3D-ResNet network, classifying and predicting tumor efficacy in tumor regions.

[0010] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound, as described above, is characterized in that: S1 includes...

[0011] S11. Decompose the three-dimensional transrectal ultrasound image volume data into two-dimensional grayscale rectal ultrasound image frames;

[0012] S12. Normalize each frame of two-dimensional grayscale rectal ultrasound image.

[0013] S13. Uniformly sample two-dimensional grayscale rectal ultrasound image frames within the 360° scanning range at different angle intervals.

[0014] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound as described above is characterized in that: the different angles in S13 include 6°, 9°, 18° or only the maximum longitudinal section frame.

[0015] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound, as described above, is characterized in that: S2 includes...

[0016] S21. Annotated data: Extract multiple frames of images for each case from the sampling data of two-dimensional grayscale rectal ultrasound image frames according to the tumor size, annotate some images, and divide the annotated data into training set images and validation set images according to a preset ratio. The remaining unannotated images are used for semi-supervised learning.

[0017] S22, Construction of a dual-branch semi-supervised segmentation network, including:

[0018] The supervised learning branch receives labeled samples. and real labeling Simultaneously process unlabeled images And generate pseudo tags. The loss of the supervised learning branch is expressed as:

[0019]

[0020] Unsupervised learning branch, a weakly enhanced version of the input unlabeled image. And use the pseudo-labels generated by the supervised learning branch for training: The loss of the unsupervised learning branch is expressed as:

[0021]

[0022] The total loss is the weighted sum of the losses from the two branches:

[0023]

[0024] in, For balance coefficient, Indicates a shared encoder. and There are two decoders;

[0025] S23. Segmentation result output: The automatic segmentation model for rectal tumors classifies the two-dimensional grayscale rectal ultrasound image pixel by pixel, outputs a tumor region segmentation mask, automatically delineates the region of interest of the tumor, and expands the boundary of the region of interest of the tumor to ensure complete coverage of the tumor region.

[0026] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound as described above is characterized in that: in S21, 6 to 40 frames of images are extracted from the volumetric data of three-dimensional transrectal ultrasound images according to the tumor size, and the labeled data are divided into training set images and validation set images in an 8:2 ratio.

[0027] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound as described above is characterized in that: the tumor contour annotation on the three-dimensional transrectal ultrasound image in S21 is obtained by manual annotation.

[0028] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound, as described above, is characterized in that: in S23, the boundary of the region of interest for the tumor expands outward by 4% from each spatial direction.

[0029] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound, as described above, is characterized in that: S3 includes...

[0030] S31. Pre-training and feature learning: The classification network backbone adopts the standard 3D-ResNet architecture and is pre-trained using a self-supervised contrastive learning algorithm on a large-scale unlabeled ultrasound image dataset of thyroid tumors and breast tumors to learn transferable robust features.

[0031] S32. Enhanced attention mechanism: A convolutional block attention module is inserted before the global average pooling layer of 3D-ResNet, and the channel attention and spatial attention sub-modules are applied sequentially to obtain the 3D-ResNet-CBAM classification network.

[0032] S33. Outputting the classification prediction of tumor treatment efficacy: The image outlining the region of interest of the tumor is preprocessed and input into the 3D-ResNet-CBAM classification network. After feature extraction, a classification layer is used for binary classification prediction. The classification network is trained using standard cross-entropy loss.

[0033]

[0034] Where C represents the number of categories, Let i be the predicted probability of category i. The true label is encoded by One-Hot; the model outputs a pCR pathological complete remission prediction score in the range of 0-1. When the score is higher than the optimal cutoff value maximized by the trained Settenton index, the patient is predicted to achieve pCR.

[0035] The method for predicting neoadjuvant therapy response in rectal cancer based on three-dimensional transrectal ultrasound, as described above, is characterized in that: image preprocessing in S33, which delineates the region of interest of the tumor, includes...

[0036] On each two-dimensional grayscale rectal ultrasound frame image, the minimum bounding rectangle of the lesion is first generated based on the lesion contour output by the automatic rectal tumor segmentation model. The width and height of the bounding rectangle are calculated, and the larger of the width and height values ​​is denoted as L.

[0037] Move the four sides of the circumscribed rectangle outward by 0.04L in the four directions of up, down, left, and right to obtain the expanded rectangle of the lesion;

[0038] The corresponding image is cropped out on the image using an outward-expanding rectangle to obtain the lesion image block; a rectangular image block is obtained by symmetrically filling the short side of the rectangular image block with pixel values ​​of 0 to obtain a square lesion image block.

[0039] To ensure uniform model input size, all image blocks were scaled to 224×224 pixels.

[0040] The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound, as described above, is characterized by setting the training parameters of the pathological complete remission prediction model based on 3D-ResNet network as follows: batch size 16, initial learning rate 0.002, training iterations 2000 times, and Dropout regularization coefficient 0.5.

[0041] The beneficial effects of this invention are:

[0042] 1. This invention is the first to achieve automated outcome prediction based on ultrasound imaging of rectal cancer. It is the first to propose using 3D TRUS data to construct a deep learning framework to predict pCR (progressive complete remission) in LARC patients after neoadjuvant therapy, filling a gap in this field. Furthermore, rectal ultrasound equipment is less expensive than MRI, easier to operate, and more suitable for resource-limited areas and primary healthcare institutions, thus having higher clinical accessibility and wider application value.

[0043] 2. This invention achieves automated segmentation, reducing reliance on manual intervention. In this invention, the semi-supervised segmentation model is highly consistent with expert manual annotation (Dice score 0.89), significantly outperforming the standard U-Net (Dice score 0.86, Intersection over Union (IoU) 0.77, P < 0.001). Furthermore, the segmentation performance remains stable across tumor subgroups with different T stages (T1-2 vs T3 vs T4), sizes (<4.0 cm vs ≥4.0 cm), and locations (distance from the anal verge <5.0 cm vs ≥5.0 cm), effectively solving the problems of excessive time consumption and high subjectivity associated with manual segmentation, laying the foundation for building a large-scale database.

[0044] 3. This invention exhibits high predictive accuracy, aiding in precise treatment decisions. The invention was trained and evaluated on a dataset comprising 538 multicenter patients (348 in the training set, 87 in the internal validation set, and 103 in the external validation set). Using 3D-ResNet prediction models with 9° or 18° sampling intervals, the area under the receiver operating characteristic (AUC) reached 0.92 in the internal validation set and 0.85-0.86 in the external multicenter validation set, with accuracies of 86.2%-90.8% and 81.6%-83.5%, respectively. Calibration curves showed good agreement between predicted probabilities and actual outcomes (P > 0.05), and decision curve analysis demonstrated a clinically meaningful net benefit, providing a basis for patient selection using a "wait and see" strategy.

[0045] 4. The sampling strategy optimization of this invention improves generalization ability. This invention finds that the prediction performance obtained by sampling intervals of 9° and 18° is better than that of overly dense (6°) or overly sparse (single frame) sampling. While balancing information density and computational complexity, it improves the generalization ability of the model and provides a methodological reference for the development of AI models based on 3D ultrasound data. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the invention, where A represents data preprocessing; B represents a semi-supervised segmentation dual-branch network; and C represents a 3D-ResNet prediction network. Detailed Implementation

[0047] The technical solutions in the embodiments of the present invention will now be clearly and completely described in conjunction with the accompanying drawings.

[0048] In this invention, descriptions involving terms such as "preferred" and "second-best" are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "preferred" or "second-best" may explicitly or implicitly include at least one of those features.

[0049] This invention was supported by the National Natural Science Foundation of China (Grant No. 82320108011).

[0050] like Figure 1 As shown, a method for predicting neoadjuvant therapy response in rectal cancer based on three-dimensional transrectal ultrasound includes the following steps: preprocessing of 3D TRUS three-dimensional transrectal ultrasound image volume data, automatic tumor segmentation based on semi-supervised learning, and prediction of pCR pathological complete remission based on 3D-ResNet.

[0051] S1. Preprocessing of three-dimensional transrectal ultrasound image volume data to obtain sampling data of two-dimensional grayscale rectal ultrasound image frames within the scanning range; specifically including...

[0052] S11. Decompose the three-dimensional transrectal ultrasound image volume data into two-dimensional grayscale TRUS rectal ultrasound image frames.

[0053] S12. Normalize each frame of image to eliminate grayscale differences between different patients;

[0054] S13. Uniformly sample two-dimensional grayscale rectal ultrasound image frames within the 360° scanning range at different angle intervals; among which, uniform sampling is performed at different angle intervals such as 6°, 9°, 18° or only the single frame of the largest longitudinal section.

[0055] S2. Automatic segmentation of rectal tumors based on semi-supervised learning, which segments two-dimensional frame sampled data and outputs an image outlining the region of interest of the tumor;

[0056] To reduce the workload of medical image annotation and improve segmentation accuracy, this invention employs a semi-supervised learning (SSL) method to construct a tumor segmentation model. Specifically, it includes...

[0057] S21. Data preparation: 6 to 40 images per case are uniformly extracted from the sampled data of two-dimensional grayscale rectal ultrasound images, based on tumor size. Experienced radiologists annotate the tumor contours of some images. The annotated data is divided into training and validation sets in an 8:2 ratio, with the remaining unannotated data used for semi-supervised learning. Furthermore, this invention eliminates the need for subsequent manual annotation during application.

[0058] S22. Construction of a dual-branch semi-supervised segmentation network: A dual-branch semi-supervised learning (SSL) framework is constructed, comprising a shared encoder and two task-specific decoders. Specifically, it includes:

[0059] Branch 1 (Supervised Learning Branch): Receives labeled samples and accurate labeling Segmentation is learned under the supervision of real-labeled images. The supervised learning branch also processes unlabeled images. And generate pseudo tags. The loss of the supervised learning branch is expressed as:

[0060]

[0061] Branch 2 (Unsupervised learning branch): A weakly enhanced version of the input unlabeled image. And use the pseudo-labels generated by branch one for training: The loss of the unsupervised learning branch is expressed as:

[0062]

[0063] The total loss is the weighted sum of the losses from the two branches:

[0064]

[0065] in, The balance coefficient (optimized value) 2) indicates a shared encoder. and With two decoders, this design enables the sharing of feature representations while also enabling differentiated learning of labeled and unlabeled data.

[0066] S23. Segmentation result output: The model performs pixel-by-pixel classification on the sampled data of the two-dimensional grayscale rectal ultrasound image frame after S1 preprocessing, outputs a tumor region segmentation mask, automatically delineates the tumor region of interest (ROI), and expands the ROI boundary outward (generally set to expand outward by 4% in each spatial direction) to ensure complete coverage of the tumor region, thereby reducing the impact of segmentation error on subsequent prediction.

[0067] S3. A pathological complete remission (pCR) prediction model based on a 3D-ResNet network, specifically including:

[0068] S31. Pre-training and Feature Learning: The classification network backbone adopts the standard 3D-ResNet architecture. First, it is pre-trained using a self-supervised contrastive learning algorithm on a large-scale unlabeled public ultrasound image dataset of thyroid tumors and breast tumors to learn transferable robust features and enhance the model's generalization ability on multi-center data.

[0069] S32. Enhanced attention mechanism: A convolutional block attention module (CBAM) is inserted before the global average pooling layer of 3D-ResNet. The channel attention and spatial attention sub-modules are applied sequentially, and the combination of the two enhances the model's sensitivity to prognostic regions.

[0070] S33. Output the classification prediction of tumor efficacy. The image delineating the region of interest (ROI) of the tumor is preprocessed, i.e., cropped from the original 3D volume and resampled to a 224×224 ROI image patch in step S23, and input into the 3D-ResNet-CBAM classification network. After feature extraction, a classification layer is used for binary classification prediction (whether the efficacy after neoadjuvant therapy reaches pCR or not). The classification network is trained using standard cross-entropy loss.

[0071]

[0072] Where C represents the number of categories (C=2). Let i be the predicted probability of category i. The true labels are One-Hot encoded. The training parameters in this invention are set as follows: batch size 16, initial learning rate 0.002, training iterations 2000, and Dropout regularization coefficient 0.5.

[0073] The image preprocessing in S33 for outlining the region of interest of the tumor includes the following steps: First, on each two-dimensional grayscale rectal ultrasound frame image, the minimum bounding rectangle of the lesion is generated based on the lesion contour output by the automatic rectal tumor segmentation model. The width and height of the bounding rectangle are calculated, and the larger of the width and height values ​​is denoted as L. The four sides of the bounding rectangle are moved outward by 0.04L in the four directions (up, down, left, and right) to obtain the outer rectangle of the lesion. The corresponding image is cropped out of the image using the outer rectangle to obtain the lesion image block. The cropped image block is usually rectangular. The short side of the rectangular image block is symmetrically filled with pixel values ​​of 0 to obtain a square lesion image block. To ensure the uniformity of the model input size, all image blocks are scaled to 224×224 pixels.

[0074] Finally, the prediction model outputs a pCR prediction score in the range of 0-1. When the score is higher than the optimal cutoff value maximized by the trained Setuden index (for example, the optimal cutoff value is 0.485 for a 9° sampling interval), it is predicted that the patient will achieve pCR, thereby assisting clinical decision-making.

[0075] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct or indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound, characterized in that: Including S1. Preprocessing of three-dimensional transrectal ultrasound image volume data to obtain sampling data of two-dimensional grayscale rectal ultrasound image frames within the scanning range; S2. Automatic segmentation of rectal tumors based on semi-supervised learning, which segments two-dimensional frame sampled data and outputs an image outlining the region of interest of the tumor; S3. Pathological complete remission prediction based on 3D-ResNet network, classifying and predicting tumor efficacy in tumor regions.

2. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 1, characterized in that: S1 includes S11. Decompose the three-dimensional transrectal ultrasound image volume data into two-dimensional grayscale rectal ultrasound image frames; S12. Normalize each frame of two-dimensional grayscale rectal ultrasound image. S13. Uniformly sample two-dimensional grayscale rectal ultrasound image frames within the 360° scanning range at different angle intervals.

3. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 2, characterized in that: The different angles in S13 include 6°, 9°, 18°, or only the maximum longitudinal section of a single frame.

4. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 1, characterized in that: S2 includes S21. Annotated data: Extract multiple frames of each case from the sampling data of two-dimensional grayscale rectal ultrasound image frames according to the tumor size, annotate some images, and divide the annotated data into training set images and validation set images according to a preset ratio. The remaining unannotated images are used for semi-supervised learning. S22, Construction of a dual-branch semi-supervised segmentation network, including: The supervised learning branch receives labeled samples. and real labeling Simultaneously process unlabeled images And generate pseudo tags. The loss of the supervised learning branch is expressed as: ； Unsupervised learning branch, a weakly enhanced version of the input unlabeled image. And use the pseudo-labels generated by the supervised learning branch for training: The loss of the unsupervised learning branch is expressed as: ； The total loss is the weighted sum of the losses from the two branches: ； in, The balance coefficient indicates a shared encoder. and There are two decoders; S23. Segmentation result output: The automatic segmentation model for rectal tumors classifies the two-dimensional grayscale rectal ultrasound image pixel by pixel, outputs a tumor region segmentation mask, automatically delineates the region of interest of the tumor, and expands the boundary of the region of interest of the tumor to ensure complete coverage of the tumor region. 5.S23. Segmentation result output: The automatic segmentation model for rectal tumors classifies the two-dimensional grayscale rectal ultrasound image pixel by pixel, outputs a tumor region segmentation mask, automatically delineates the region of interest of the tumor, and expands the boundary of the region of interest of the tumor to ensure complete coverage of the tumor region.

6. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 5, characterized in that: The tumor contour annotation on the three-dimensional transrectal ultrasound image in S21 was obtained through manual annotation.

7. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 4, characterized in that: In S23, the boundaries of the region of interest for the tumor expand outward by 4% in each spatial direction.

8. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 1, characterized in that: S3 includes S31. Pre-training and feature learning: The classification network backbone adopts the standard 3D-ResNet architecture and is pre-trained using a self-supervised contrastive learning algorithm on a large-scale unlabeled ultrasound image dataset of thyroid tumors and breast tumors to learn transferable robust features. S32. Enhanced attention mechanism: A convolutional block attention module is inserted before the global average pooling layer of 3D-ResNet, and the channel attention and spatial attention sub-modules are applied sequentially to obtain the 3D-ResNet-CBAM classification network. S33. Outputting the classification prediction of tumor treatment efficacy: The image outlining the region of interest of the tumor is preprocessed and input into the 3D-ResNet-CBAM classification network. After feature extraction, a classification layer is used for binary classification prediction. The classification network is trained using standard cross-entropy loss. ； Where C represents the number of categories, Let i be the predicted probability of category i. The true label is encoded by One-Hot; the model outputs a pCR pathological complete remission prediction score in the range of 0-1. When the score is higher than the optimal cutoff value maximized by the trained Settenton index, the patient is predicted to achieve pCR.

9. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 8, characterized in that: Image preprocessing for outlining the region of interest for the tumor in S33 includes: On each two-dimensional grayscale rectal ultrasound frame image, the minimum bounding rectangle of the lesion is first generated based on the lesion contour output by the automatic rectal tumor segmentation model. The width and height of the bounding rectangle are calculated, and the larger of the width and height values ​​is denoted as L. Move the four sides of the circumscribed rectangle outward by 0.04L in the four directions of up, down, left, and right to obtain the expanded rectangle of the lesion; The corresponding image is cropped out on the image using an outward-expanding rectangle to obtain the lesion image block; a rectangular image block is obtained by symmetrically filling the short side of the rectangular image block with pixel values ​​of 0 to obtain a square lesion image block. To ensure uniform model input size, all image blocks were scaled to 224×224 pixels.

10. The method for predicting the response to neoadjuvant therapy for rectal cancer based on three-dimensional transrectal ultrasound according to claim 8, characterized in that: The training parameters for the pathological complete remission prediction model based on the 3D-ResNet network were set as follows: batch size 16, initial learning rate 0.002, training iterations 2000, and Dropout regularization coefficient 0.5.

Images