A cervical cancer radiotherapy patient osteoporosis risk prediction system and method based on a transformer

Abstract

The application provides a cervical cancer radiotherapy patient osteoporosis risk prediction system and method based on a Transformer, and belongs to the cross field of artificial intelligence and medical imaging. The system comprises an image deep feature extraction module, an image feature self-adaptive fusion module and a multi-modal fusion prediction module. First, the pelvic baseline CT is subjected to 2.5D slice processing, and is input into a DenseNet201 to extract initial deep features. Then, a Transformer encoder is used for self-adaptive fusion to obtain a unified image feature vector. Finally, the image feature vector is spliced with key clinical features and input into a prediction model to obtain a risk result. Experiments show that the method has excellent prediction performance, with an external validation AUC of up to 0.977, and realizes accurate early evaluation of osteoporosis risk.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and medical image processing technology, specifically relating to a deep learning system and method for medical auxiliary diagnosis. More specifically, it relates to a system and method based on the Transformer architecture, which integrates multimodal medical image features and clinical features to predict the risk of osteoporosis in cervical cancer patients after receiving external pelvic radiotherapy. Background Technology

[0002] Cervical cancer is a common malignant tumor that seriously threatens women's health. Radiotherapy is one of the core methods for radical treatment of locally advanced cervical cancer. However, when pelvic radiotherapy irradiates the tumor target area, it inevitably distributes high doses of radiation to the pelvic bones, leading to bone tissue damage, decreased bone density, osteoporosis, and even pathological fractures. These skeletal-related adverse events significantly reduce patients' long-term quality of life.

[0003] Currently, clinical monitoring of bone injury after radiotherapy mainly relies on regular follow-up imaging (such as computed tomography CT scans). This is a passive, post-treatment diagnostic model that cannot achieve early warning and intervention. Traditional risk assessment models often rely on statistical analysis of limited clinical factors (such as age and menopausal status), resulting in limited predictive efficacy and failing to fully utilize the rich anatomical and textural information contained in baseline CT images before treatment.

[0004] In recent years, deep learning-based medical image analysis technology has made significant progress. Convolutional neural networks can automatically extract deep features from images, but the feature representation ability of a single network model has limitations, and the features extracted by different network architectures are complementary. The Transformer architecture, due to its powerful global context modeling capabilities and parallel processing advantages, has shown potential in feature fusion tasks. How to effectively and adaptively fuse heterogeneous deep image features with key clinical dosimetry parameters to build a high-precision, highly generalizable predictive model remains a challenge in current clinical prediction research.

[0005] Currently, there is no publicly available solution that systematically applies technologies such as 2.5D medical image processing, parallel feature extraction using multi-heterogeneous convolutional neural networks, Transformer attention fusion, and deep integration of image and clinical multimodal approaches to predicting osteoporosis risk after cervical cancer radiotherapy. Therefore, developing a system capable of personalized, high-precision risk prediction before radiotherapy is of urgent clinical need and significant application value. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a Transformer-based system and method for predicting the risk of osteoporosis in cervical cancer patients undergoing radiotherapy. This system and method, through an innovative multimodal feature fusion framework, aims to achieve high-precision prediction of the risk of osteoporosis in patients before radiotherapy, providing decision support for early clinical intervention and the development of personalized treatment plans.

[0007] This invention is achieved through the following technical solution:

[0008] On the one hand, the present invention provides a Transformer-based osteoporosis risk prediction system for cervical cancer radiotherapy patients, which includes: an image depth feature extraction module, an image feature adaptive fusion module, and a multimodal fusion prediction module.

[0009] Image depth feature extraction module: Used to extract depth features from preprocessed 2.5D medical image slices using convolutional neural networks DenseNet201 and ResNet50, including a 2.5D image preprocessing unit and a parallel depth feature extraction unit.

[0010] 2.5D Image Preprocessing Unit: Acquires the patient's pelvic baseline 3D CT image data, locates the slice with the largest cross-sectional area in the region of interest defined by the pelvic bone contour as the target slice, and selects axial slices of a specific number of adjacent layers (such as ±1, ±2, ±4 layers) centered on the target slice to form a 2.5D slice group.

[0011] Parallel deep feature extraction unit: 2.5D slices are simultaneously input into multiple pre-trained convolutional neural networks with heterogeneous structures. These networks may include, but are not limited to, ResNet50, DenseNet201, etc., with each network working independently to extract high-dimensional deep features.

[0012] Image feature adaptive fusion module: connected to the image depth feature extraction module, used to fuse the depth features output by the convolutional neural network using a Transformer encoder based on a multi-head self-attention mechanism to obtain a unified image depth feature vector.

[0013] Transformer Feature Fusion: Deep features extracted from different convolutional neural networks are concatenated and input into a Transformer encoder. The Transformer encoder includes a positional encoding layer, a multi-head self-attention layer, a feedforward neural network layer, and residual connections and layer normalization layers. These layers reweight and fuse the initial deep feature sequence after concatenation, outputting a fused image depth feature vector. Through its core multi-head self-attention mechanism, the encoder adaptively learns and fuses features from different sources, weighing their importance, and ultimately outputs a robust, fused image depth feature vector.

[0014] Multimodal fusion prediction module: connected to the image feature adaptive fusion module, used to concatenate the unified image depth feature vector with the clinical feature vector, and output osteoporosis risk prediction results through a prediction classifier.

[0015] Feature stitching unit: used to stitch the fused image depth feature vector with the clinical feature vector extracted from the patient's medical record to form a multimodal joint feature vector; the clinical feature vector includes at least the patient's age, the percentage of lumbosacral bone irradiated with V10Gy, the percentage of pelvic bone irradiated with V20Gy, and the percentage of pelvic bone irradiated with V40Gy.

[0016] The classification and prediction unit is used to input the multimodal joint feature vector into the final prediction classifier and output the risk prediction result of the patient developing osteoporosis after radiotherapy. The final prediction classifier is a logistic regression model or a decision tree ensemble model. This classifier outputs a risk probability value to assess the patient's risk level of developing osteoporosis.

[0017] On the other hand, the present invention also provides corresponding methods, electronic devices, and computer-readable storage media.

[0018] A Transformer-based method for predicting osteoporosis risk in cervical cancer patients undergoing radiotherapy, comprising the following steps:

[0019] Step S1: Collect baseline pelvic CT images and clinical data including age and radiation dose parameters of the patient;

[0020] Step S2: Perform 2.5D preprocessing on the CT images to obtain a 2.5D slice group; the 2.5D preprocessing is as follows: take the pelvic bone contour as the region of interest, locate the slice where its largest cross-section is located as the target slice, and extract a total of seven continuous axial slices, including the target slice and its adjacent ±1, ±2 and ±4 layers, with the target slice as the center, to form a 2.5D slice group.

[0021] Step S3: Use two heterogeneous pre-trained convolutional neural networks to perform parallel depth feature extraction on the 2.5D slice group to obtain initial depth features; the convolutional neural networks used include ResNet50 and DenseNet201.

[0022] Step S4: Adaptively fuse the initial depth features using a Transformer encoder to obtain a unified image depth feature vector;

[0023] Step S5: Concatenate the image depth feature vector with the clinical feature vector to form a multimodal joint feature vector;

[0024] Step S6: Input the multimodal joint feature vector into the final predictive classifier and output the osteoporosis risk probability;

[0025] Step S7: Based on the risk probability, classify the patient into risk groups.

[0026] Furthermore, to verify the superiority and necessity of the proposed multimodal fusion and prediction module architecture, several alternative technical paths were designed and implemented for comparative verification during the construction of this system. These paths include: a path that uses only the image depth feature vectors fused by the Transformer for prediction (Path 1); a path that uses an ensemble learning method to aggregate the slice-level prediction results of the parallel convolutional neural network (Path 2); a path that uses a multi-instance learning method to reorganize and fuse the features extracted in parallel (Path 3); and the core of this invention, the multimodal fusion path of image and clinical features as described in steps B1) and B2) above (Path 4). Experimental results on multicenter retrospective data show that the prediction model corresponding to Path 4 achieves significantly better prediction performance on both internal and external validation sets, with key indicators such as the area under the curve (AUC) consistently leading, especially in external independent validation where the AUC can reach as high as 0.977, fully demonstrating the effectiveness of the multimodal fusion architecture proposed in this invention.

[0027] The beneficial effects of this invention are as follows:

[0028] 1. High prediction accuracy and robustness: This method preserves the 3D spatial context through 2.5D processing, extracts complementary features using multi-heterogeneous CNNs, and then intelligently fuses these features using a Transformer. Finally, it combines these features with key clinical features to construct a prediction model that fully utilizes information. Systematic comparative experiments demonstrate that this multimodal fusion approach significantly outperforms other mainstream feature fusion methods.

[0029] 2. Strong generalization and interpretability: The system framework is general and does not depend on a single network. Preliminary transfer validation shows that the model can be applied to predict bone injury risk in other pelvic radiotherapy diseases (such as endometrial cancer). Furthermore, the Transformer's attention mechanism provides the possibility of visualizing feature importance, enhancing clinical credibility.

[0030] 3. Outstanding clinical value: The system can provide individualized risk assessment before radiotherapy planning, enabling clinicians to identify high-risk patients in advance and consider implementing personalized intervention measures such as bone marrow protective intensity-modulated radiotherapy and drug prophylaxis, thus promoting the transformation of the treatment model towards "proactive prevention". Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the overall system workflow of one embodiment of the present invention. Detailed Implementation

[0032] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0033] This invention aims to address the challenge of early prediction of osteoporosis in cervical cancer patients after radiotherapy. The following example, using a cervical cancer patient treated at the First Affiliated Hospital of China Medical University, demonstrates this approach. Figure 1 The system module diagram shown illustrates the implementation process of this method in detail.

[0034] The main modules in the method include:

[0035] 1. Data Input and 2.5D Preprocessing Module: This module receives DICOM format CT images and structured clinical data from patients and generates 2.5D slice sets.

[0036] 2. Multi-CNN Parallel Feature Extraction Engine: It has multiple pre-trained heterogeneous CNN models built in for parallel computation of deep features.

[0037] 3. Transformer Feature Fuder: Enables deep feature fusion based on multi-head self-attention.

[0038] 4. Multimodal fusion and prediction core: responsible for stitching together imaging features and clinical features and calculating the final risk.

[0039] The steps are as follows:

[0040] Step 1: After the patient completes the pre-radiotherapy localization CT scan, the system retrieves the patient's baseline pelvic CT image from the radiotherapy planning system. Simultaneously, the system obtains the patient's age from the hospital information system and extracts key dosimetric parameters from the radiotherapy planning system, such as V10Gy for the lumbosacral region and V20Gy and V40Gy for the pelvic region.

[0041] Step 2: In the 2.5D preprocessing module, the system automatically loads the CT images and, within the outlined pelvic bone contour, calculates the layer with the largest area among all axial slices, identifying it as the central target layer. Subsequently, the system automatically extracts this central layer along with its preceding layers 4, 2, and 1, and its posterior layers 1, 2, and 4, totaling seven consecutive CT slices. These slices are then normalized in grayscale (e.g., mapped to [-1,1]) and resampled to a uniform size (e.g., 224x224 pixels) to form a standard 2.5D input sample.

[0042] Step 3: In the parallel feature extraction engine, the samples composed of the above seven slices are simultaneously input into three pre-trained CNN models: ResNet50, DenseNet121, and DenseNet201. Each network performs forward propagation independently, and extracts a high-dimensional deep feature vector before its final global pooling layer (e.g., each network outputs a 1024-dimensional feature).

[0043] Step 4: In the Transformer feature fusion unit, the three feature vectors obtained in Step 3 are concatenated to form a longer joint feature vector (e.g., 3072 dimensions). This vector sequence is then input into a multi-layer Transformer encoder. The encoder uses a multi-head self-attention mechanism to dynamically calculate and weight the correlations between different feature elements, thereby automatically focusing on the most discriminative information. Finally, the encoder output is aggregated (e.g., by average pooling) to generate a fixed-dimensional, fused image depth feature vector.

[0044] Step 5: In the multimodal fusion and prediction core, the image depth feature vector obtained in step 4 is directly concatenated with the clinical feature vector [age, LSS_V10, Pelvis_V20, Pelvis_V40] prepared in step 1 to form the final multimodal joint feature vector.

[0045] Step 6: Input the multimodal joint feature vector into a pre-trained final classifier (e.g., a logistic regression model). The model outputs a risk probability value P between 0 and 1. The system can preset a clinical threshold (e.g., 0.5). If P ≥ 0.5, the patient is determined to be "high-risk for osteoporosis"; otherwise, "low-risk".

[0046] Step 7: The system generates a structured report, including risk level, probability value, and contribution analysis of key features, and sends it to the attending physician. For high-risk patients, the report can include clinical recommendations, such as: "It is recommended to assess the necessity of using pelvic bone marrow protective intensity-modulated radiotherapy and consider baseline bone mineral density monitoring and prophylactic medication."

[0047] Model training and validation instructions:

[0048] The training of this system model was based on historical data. Cases of cervical cancer patients who were pathologically diagnosed, received pelvic radiotherapy, and had complete follow-up CT data were collected. The mean Henlein unit value of the pelvic bone measured by CT scan 9-12 months after radiotherapy was used as the diagnostic criterion for osteoporosis (gold standard).

[0049] The training process includes the following key steps:

[0050] 1. Data Splitting: Data from one center is randomly split into a training set and an internal validation set in a 7:3 ratio. Data from two other independent centers is collected as an external validation set to test generalization ability.

[0051] 2. Multi-path comparison and verification: To verify the superiority of the multimodal fusion architecture of this invention, multiple prediction paths were simultaneously constructed and compared during training:

[0052] Path 1 (Image Only): Prediction is made using only the Transformer fused image features output from Step 4.

[0053] Path 2 (Traditional Ensemble): Use multiple CNNs to independently predict the seven slices, and then take the maximum or average of the predicted probabilities as the final result.

[0054] Path 3 (Multi-instance learning): Organize the features extracted by CNN into "bags", generate features using methods such as bag-of-words model, and then use machine learning models for prediction.

[0055] Path 4 (Invention): Completely execute the multimodal fusion path from step 1 to step 6.

[0056] 3. Performance Evaluation: Evaluation was conducted using metrics such as area under the receiver operating characteristic (ROC) curve, accuracy, sensitivity, and specificity. Experimental results show that Path 4 (this invention) achieved the best and most stable predictive performance on all validation sets, especially in external validation where the AUC reached a maximum of 0.977, significantly outperforming other comparative paths and demonstrating the effectiveness of multimodal deep fusion.

[0057] 4. Interpretability and Application Expansion: By visualizing the attention weights of the Transformer, the source of features the model focuses on during fusion can be understood. Furthermore, preliminary studies show that applying this system to other gynecological malignancies (such as endometrial cancer) undergoing pelvic radiotherapy also demonstrates good predictive efficacy, indicating the potential for generalization of this method.

[0058] In summary, this invention achieves accurate and early prediction of osteoporosis risk in cervical cancer radiotherapy patients through an integrated multimodal deep learning framework. This system is not only a technological innovation but also a practical tool that can be seamlessly integrated into clinical workflows, facilitating precision radiotherapy and personalized management.

Claims

1. A Transformer-based system for predicting osteoporosis risk in cervical cancer patients undergoing radiotherapy, characterized in that, It includes an image depth feature extraction module, an image feature adaptive fusion module, and a multimodal fusion prediction module; Image depth feature extraction module: used to extract depth features from preprocessed 2.5D medical image slices using convolutional neural networks DenseNet201 and ResNet50; Image feature adaptive fusion module: connected to the image depth feature extraction module, used to fuse the depth features output by the convolutional neural network using a Transformer encoder based on a multi-head self-attention mechanism to obtain a unified image depth feature vector; Multimodal fusion prediction module: connected to the image feature adaptive fusion module, used to concatenate the unified image depth feature vector with the clinical feature vector, and output the osteoporosis risk prediction result through the prediction classifier.

2. The Transformer-based osteoporosis risk prediction system for cervical cancer radiotherapy patients according to claim 1, characterized in that, The image depth feature extraction module includes a 2.5D image preprocessing unit and a parallel depth feature extraction unit; Among them, the 2.5D image preprocessing unit is used to read the patient's pelvic baseline three-dimensional CT image data, take the pelvic bone contour as the region of interest, locate the slice where its largest cross-section is located as the target slice, and extract a total of seven continuous axial slices, including the target slice and its adjacent ±1, ±2 and ±4 layers, with the target slice as the center, to form a 2.5D slice group. Parallel depth feature extraction unit: used to input the 2.5D slice group into DenseNet201 and ResNet50 pre-trained convolutional neural networks, and each network independently extracts the corresponding image depth features.

3. The Transformer-based osteoporosis risk prediction system for cervical cancer radiotherapy patients according to claim 1, characterized in that, The Transformer encoder in the image feature adaptive fusion module includes a position encoding layer, a multi-head self-attention layer, a feedforward neural network layer, and a residual connection and layer normalization layer, which are used to reweight and fuse the initial depth feature sequence after stitching, and output a fused image depth feature vector.

4. The Transformer-based osteoporosis risk prediction system for cervical cancer radiotherapy patients according to claim 1, characterized in that, The multimodal fusion prediction module includes a feature splicing unit and a classification prediction unit; Feature stitching unit: used to stitch the fused image depth feature vector with the clinical feature vector extracted from the patient's medical record to form a multimodal joint feature vector; the clinical feature vector includes the patient's age, percentage of lumbosacral bone V10Gy irradiated volume, percentage of pelvic bone V20Gy irradiated volume, and percentage of pelvic bone V40Gy irradiated volume. Classification and prediction unit: used to input the multimodal joint feature vector into the final prediction classifier and output the risk prediction result of patients developing osteoporosis after radiotherapy; the final prediction classifier is a logistic regression model or a decision tree ensemble model.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the osteoporosis risk prediction steps defined by the system as described in any one of claims 1 to 4.

6. A Transformer-based method for predicting osteoporosis risk in cervical cancer patients undergoing radiotherapy, characterized in that, Applied to the system as described in any one of claims 1-4, comprising the following steps: Step S1: Collect baseline pelvic CT images and clinical data including age and radiation dose parameters of the patient; Step S2: Perform 2.5D preprocessing on the CT images to obtain 2.5D slice groups; Step S3: Use two heterogeneous pre-trained convolutional neural networks to perform parallel depth feature extraction on the 2.5D slice group to obtain initial depth features; Step S4: Adaptively fuse the initial depth features using a Transformer encoder to obtain a unified image depth feature vector; Step S5: Concatenate the image depth feature vector with the clinical feature vector to form a multimodal joint feature vector; Step S6: Input the multimodal joint feature vector into the final predictive classifier and output the osteoporosis risk probability; Step S7: Based on the risk probability, classify the patient into risk groups.

7. A method for predicting osteoporosis risk in cervical cancer radiotherapy patients based on Transformer, as described in claim 6, characterized in that... The 2.5D preprocessing in step S2 specifically involves: taking the pelvic bone contour as the region of interest, locating the slice containing its largest cross-section as the target slice, and extracting a total of seven continuous axial slices, including the target slice and its adjacent ±1, ±2, and ±4 layers, centered on the target slice, to form a 2.5D slice group.

8. A method for predicting osteoporosis risk in cervical cancer radiotherapy patients based on Transformer, as described in claim 6, characterized in that... The convolutional neural networks used in step S3 include ResNet50 and DenseNet201.

9. A method for predicting osteoporosis risk in cervical cancer radiotherapy patients based on Transformer, as described in claim 6, characterized in that... The method is used to predict the risk of osteoporosis in other gynecological malignant tumor patients who receive pelvic radiotherapy.

Images