Fracture risk early warning system for multiple myeloma patient based on deep learning

Abstract

The invention relates to the technical field of medical artificial intelligence and blood tumor clinical crossing, in particular to a multiple myeloma patient fracture risk early warning system based on deep learning, which comprises a data acquisition module, a multi-modal fusion module, a prediction model module, an interpretability module and an early warning push module. The data acquisition module acquires dynamic multi-modal data; after the features are extracted by the multi-modal fusion module, dimensions are unified through a full connection layer, and weight fusion is allocated according to pathological association; the prediction model module outputs the fracture risk probability of the patient in the next six months through a Transform prediction model, and optimizes the prediction model by taking fracture risk prediction as a main task and bone metabolism deterioration trend prediction as an auxiliary task; the interpretability module quantifies the contribution degree of each dynamic feature to a risk prediction result, and generates a text report of association between the change of each dynamic feature and the risk; and the early warning pushing module generates early warning information according to the fracture risk probability, and pushes the early warning information and the text report to the user terminal after grading and labeling the early warning information and the text report.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence and hematological oncology clinical technology, specifically to a fracture risk warning system for multiple myeloma patients based on deep learning. Background Technology

[0002] Multiple myeloma (MM) is a malignant hematologic disorder characterized by the abnormal proliferation of plasma cells in the bone marrow. Fractures are its most common bone-related complication, which can occur in multiple bone sites such as the spine, ribs, femur, and humerus. This complication not only severely reduces the patient's quality of life (such as long-term pain and limited mobility) but also increases the patient's risk of death.

[0003] Current technologies exist for predicting fracture risk in MM patients, but they suffer from the following problems: First, existing models often rely on "single baseline data" (such as initial CT images and fixed clinical indicators) to construct static prediction models, failing to capture the dynamic evolution of MM patients' conditions (such as the progressive increase in osteolytic lesions after chemotherapy and the periodic fluctuations in bone metabolism indicators such as serum calcium). This leads to decreased mid- to long-term (3-6 months) early warning accuracy, making it difficult to meet long-term monitoring needs. Furthermore, static models may miss MM patients, potentially causing them to miss the early intervention window (such as the optimal intervention time being 1-2 months after the onset of bone marrow edema). Second, existing technologies often use a simple splicing and fusion of "clinical indicators" and "single images" as features, failing to explore the dynamic correlations between different modalities of data (such as the relationship between the increase in vertebral osteolytic lesions and serum calcium). The pathological association of elevated microglobulin levels leads to insufficient utilization of multimodal features, failing to fully reflect the complex pathological mechanisms of MM bone disease (such as the imbalance between bone resorption and bone formation). Thirdly, most existing prediction models only output risk probabilities without explaining pathological associations, making it difficult for physicians to determine the "source of risk." Warning results may be abandoned clinically because the cause of the risk cannot be traced, requiring expert physicians to rely on experience for further judgment. Therefore, there is an urgent need for a deep learning system that can integrate dynamic data and deeply fuse multimodal features to improve the accuracy and clinical applicability of fracture risk prediction in multiple myeloma patients. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a fracture risk early warning system for multiple myeloma patients based on deep learning, which can improve the accuracy and clinical applicability of fracture risk prediction for multiple myeloma patients.

[0005] The basic solution provided by this invention is a fracture risk early warning system for multiple myeloma patients based on deep learning, including a data acquisition module, a multimodal fusion module, a prediction model module, an interpretability module, and an early warning push module; The data acquisition module is used to collect dynamic multimodal data from patients with multiple myeloma. This dynamic multimodal data includes dynamic clinical data, dynamic imaging data, and dynamic laboratory data. The dynamic clinical data includes disease staging, pain scores, and records of bone-targeted drug use. The dynamic imaging data includes three-dimensional CT images of the skeleton and MRI images of bone marrow edema. The dynamic laboratory data includes serum calcium and serum... - Microglobulins, alkaline phosphatase; A multimodal fusion module is used for feature extraction and fusion of dynamic multimodal data. This module includes an image feature extraction submodule, a temporal feature extraction submodule, and a feature fusion submodule. The image feature extraction submodule processes 3D CT images to extract features related to osteolytic lesions and processes MRI images to extract features related to bone marrow edema, outputting an image feature vector. The temporal feature extraction submodule processes dynamic clinical data and dynamic laboratory data, capturing trend changes in indicators and outputting a temporal feature vector. The feature fusion submodule maps the image feature vector and the temporal feature vector to the same dimension using a fully connected layer, determines the association weights of the two types of features using a pre-defined multiple myeloma bone disease pathology association rule base, and performs feature fusion using the following formula, outputting a multimodal fused feature vector:

[0006] in, This is a multimodal fusion feature vector. For image feature vectors, For time series feature vectors, For association weights; A prediction model module is used to predict fracture risk based on multimodal fusion feature vectors. The prediction model module includes a Transformer prediction model and a dual-task training submodule. The Transformer prediction model takes the multimodal fusion feature vectors as input and outputs the probability of fracture risk for the patient within a preset time period. The dual-task training submodule takes fracture risk prediction as the main task and bone metabolism deterioration trend prediction as the secondary task. It optimizes the Transformer prediction model through a multi-task loss function to improve the model's generalization ability and prediction accuracy for fracture risk prediction. The multi-task loss function is obtained by weighted averaging of the cross-entropy loss function of the main task and the mean squared error loss function of the secondary task. An interpretability module is used to parse the feature contribution logic of the fracture risk probability. The feature contribution logic is constructed based on the dynamic features corresponding to the dynamic clinical data, dynamic imaging data, and dynamic laboratory data. The interpretability module includes a SHAP calculation submodule and an association reporting submodule. The SHAP calculation submodule is used to calculate the contribution of each dynamic feature to the fracture risk probability. The association reporting submodule is used to generate a text report on the correlation between the changes of each dynamic feature and the fracture risk. The correlation is determined based on the pathological association rule library of multiple myeloma bone disease. The early warning push module is used to generate early warning information based on the fracture risk probability. The early warning information includes the risk level, the risk location, and intervention suggestions. The early warning information and the text report are pushed to the user terminal. The higher the fracture risk probability, the higher the risk level.

[0007] The principle behind this basic solution is as follows: By connecting to the hospital's clinical, imaging, and laboratory systems through a data acquisition module, and following the routine follow-up patterns for MM patients (regular updates of clinical / laboratory indicators and regular imaging re-examinations), it automatically collects dynamic clinical data (disease stage, pain score, and bone-targeted drug records), dynamic imaging data (3D CT of bones and MRI of bone marrow edema), and dynamic laboratory data (serum calcium, serum...). The comprehensive data on microglobulins and alkaline phosphatase captures the gradual evolution of MM disease (such as the enlargement of osteolytic lesions and periodic fluctuations in blood calcium), providing a complete multi-dimensional input for subsequent prediction.

[0008] In the multimodal fusion module, the image feature extraction submodule focuses on core bone structure information, extracting structural features such as osteolytic lesion volume and bone density heterogeneity from CT, and early injury features such as the proportion of bone marrow edema area from MRI, outputting image feature vectors. The temporal feature extraction submodule focuses on trend changes, capturing dynamic trends in clinical and laboratory indicators (such as continuously rising blood calcium and increasing pain scores), outputting temporal feature vectors to avoid the randomness of data from a single time point. The fully connected layer in the feature fusion submodule maps the image feature vectors and temporal feature vectors to the same dimension (eliminating information loss caused by dimensional heterogeneity). Then, a pre-defined multiple myeloma bone disease pathology association rule base determines the association weights of the two types of features. Image features (such as osteolytic lesions and bone marrow edema) directly reflect bone structure destruction and early injury status, and are crucial for fracture detection. The direct pathological basis of the disease (such as osteolytic lesions leading to decreased bone load-bearing capacity, directly increasing the risk of fracture) has a stronger pathological correlation and is more directly directional in predicting fracture risk. Temporal features can capture the dynamic evolution of the disease (such as continuously elevated blood calcium indicating increased bone resorption, and rising pain scores reflecting the progression of bone damage), supplementing the missing information in the "time dimension" of imaging features (such as changes in bone metabolism during the interval between imaging follow-ups). Since temporal features (such as fluctuations in blood calcium and changes in pain scores) are mostly indirect reflections of abnormal bone metabolism or disease progression, some temporal indicators may be affected by non-bone disease factors (such as short-term dietary effects on blood calcium). Therefore, when there is a pathological correlation between the two types of features, imaging features should be given higher weight to ensure that the core pathological factors dominate the prediction. If there is no pathological correlation (such as osteolytic lesions and weight changes), the correlation weights should be evenly distributed to avoid the risk of missing a single feature. The two types of features are ultimately fused through a formula to output a multimodal fusion feature vector, which can avoid the limitations of single-modal data. For example, image features alone cannot reflect early bone metabolism abnormalities (such as changes in blood calcium before the onset of bone marrow edema), and time-series features alone cannot locate specific risk sites (such as femoral and spinal lesions). The fusion of the two can fit the complete pathological chain and improve the comprehensiveness and accuracy of prediction.

[0009] In the prediction model module, the self-attention mechanism of the Transformer prediction model is used to accurately capture the complex correlations between multimodal fusion features (such as the synergistic effect of "increased lesion volume" and "elevated blood calcium"). The probability value in the 0-1 interval is output through the last fully connected layer and activation function in the Transformer prediction model, which outputs the patient's fracture risk probability in the future within a preset time period (such as 6 months), covering the monitoring needs of the near, mid and long term. The dual-task training sub-module takes "fracture risk prediction" as the main task (optimizing classification accuracy) and "bone metabolism deterioration trend prediction" as the secondary task (using the temporal changes of blood calcium and alkaline phosphatase as labels). The multi-task loss function alleviates the overfitting that may be caused by the small number of MM fracture samples and improves the generalization ability of the prediction model.

[0010] In the interpretability module, the SHAP calculation submodule quantifies the contribution of each dynamic feature to the early warning result (e.g., "the increase in vertebral osteolytic lesions contributes 35%, and the increase in blood calcium contributes 20%)", clarifying the core source of risk; the correlation report submodule combines MM bone disease pathology knowledge (e.g., "osteolytic lesions lead to a decrease in vertebral weight-bearing capacity, and the increase in blood calcium exacerbates bone resorption") to generate a structured text report, transforming the "black box" prediction into an understandable result with "feature contribution" and "pathological significance", helping doctors trace the cause of risk and increase their trust in the early warning result.

[0011] In the early warning push module, early warning information is generated based on the probability of fracture risk. The early warning information includes the risk level, the location of the risk, and intervention suggestions. The early warning information and the text report are pushed to the user terminal, forming a complete closed loop of data collection, prediction, interpretation, and intervention prompts. It can be integrated into the routine follow-up process of MM without manual intervention, reducing clinical operation costs.

[0012] The beneficial effects of this basic scheme are as follows: 1. Break through the limitations of static prediction and improve the accuracy of mid- and long-term early warning: Dynamic data collection covers the entire cycle of MM disease evolution. Combined with multimodal fusion to capture dynamic features, it can effectively identify mid- and long-term risk signals such as the enlargement of osteolytic lesions and the progression of bone marrow edema, and avoid missing the early intervention window of 1-2 months after the onset of bone marrow edema.

[0013] 2. Improve the utilization rate of multimodal features and align with pathological mechanisms: Compared with the traditional "simple splicing" fusion method, this solution improves the utilization rate of multimodal features by combining "pathological association weights" with dimension unification. Moreover, the fusion results are more in line with the core pathological logic of MM bone disease, which is "structural destruction, bone metabolism imbalance, and fracture", reducing the interference of non-specific features (such as weight changes) on risk assessment.

[0014] 3. Enhance clinical interpretability and promote the implementation of the plan: The interpretability module transforms the "black box" of deep learning prediction into clinically understandable results by quantifying feature contribution and reporting its correlation with pathology. This improves doctors' acceptance of the warning results and solves the clinical translation barrier of "the warning results cannot be traced back to their causes".

[0015] 4. Forming a closed-loop process to assist clinical decision-making: The entire process from dynamic data collection to early warning push is automated, and it can output the probability of medium and long-term risks, the analysis of risk causes and intervention suggestions. Doctors can quickly formulate intervention plans without relying on experience judgment (such as prioritizing vertebral body reinforcement surgery for high-risk patients), improve the efficiency of clinical decision-making, and provide technical support for the "early detection and early intervention" of fracture risk in MM patients.

[0016] Furthermore, the data acquisition module also includes a data preprocessing submodule. This submodule normalizes the 3D CT images and smooths the dynamic time-series data to remove outlier interference. By normalizing the 3D CT images (eliminating grayscale differences between different devices) and smoothing the dynamic time-series data (removing outliers from single measurements, such as occasional elevated blood calcium levels), the data preprocessing submodule reduces feature noise, avoids model misjudgments caused by invalid data, and ensures the authenticity of the time-series feature trends.

[0017] Furthermore, the dynamic clinical data and dynamic laboratory data are updated according to a first fixed cycle, and the dynamic imaging data are collected according to a second fixed cycle. The first fixed cycle and the second fixed cycle are adapted to the clinical follow-up pattern of multiple myeloma patients. The three-dimensional CT images adopt the bone window reconstruction mode, and the bone marrow edema sequence MRI images adopt T2 fat-suppressed images. The disease staging is the ISS staging, and the pain score is the VAS score.

[0018] Dynamic clinical and laboratory data are updated according to the first fixed cycle (matching the routine clinical cycle for follow-up of MM patients), and dynamic imaging data are collected according to the second fixed cycle (e.g., every 3 months, balancing radiation dose and bone structure change monitoring needs), ensuring the timeliness and regularity of data collection and avoiding incomplete capture of disease evolution due to chaotic collection intervals; CT uses bone window reconstruction to ensure clear imaging of osteolytic lesions, and MRI uses T2 fat-suppressed imaging to improve the identification rate of bone marrow edema. Disease staging is clearly defined as ISS staging, and pain score is VAS score (0-10 points), which can achieve standardization of data collection and improve data compatibility between different hospitals and different equipment.

[0019] Furthermore, the association weight determination rule of the feature fusion submodule is as follows: when there is a pathological association between the image feature vector and the time series feature vector, the association weight is assigned a value greater than 0.5; when there is no pathological association between the image feature vector and the time series feature vector, the association weight is assigned a value of 0.5.

[0020] Clearly define the rules for determining the associated weights to avoid subjectivity in weight allocation (such as doctors adjusting weights based on experience) and improve the reproducibility of model training. At the same time, the rules for determining the weights are consistent with the pathological logic of MM bone disease, ensuring that "structural features dominate risk judgment" and avoiding the non-specificity of time-series features (such as elevated blood calcium caused by abnormal renal function) from masking the core risk signals.

[0021] Furthermore, the feature fusion submodule also includes an attention assistance unit; the attention assistance unit fine-tunes the association weights based on the patient's individual medical history, with the fine-tuning range not exceeding 0.1.

[0022] The attention-assisted unit fine-tunes weights (within 0.1) based on individual patient history (such as previous fracture history and duration of bone-targeted drug use) to adapt to individual differences. For example, patients with a history of fractures have poorer bone structural stability, so the correlation weight can be increased by 0.05 (enhancing the dominant role of imaging features in risk prediction); similarly, patients on long-term denosumab (a bone-targeted drug) have more stable bone metabolism, and the effectiveness of temporal changes in trends in temporal features (such as serum calcium and alkaline phosphatase) in indicating fracture risk is reduced, so the correlation weight can be increased by 0.05 (enhancing the weight of imaging features and balancing the impact of decreased effectiveness of temporal features); and further, the serum levels of patients with chronic kidney disease... - The temporal changes in microglobulin not only reflect the condition of MM, but are also directly related to the interference of renal function on bone metabolism (the risk of occult bone resorption). At this time, the effectiveness of temporal features in indicating fracture risk is significantly improved, and the correlation weight can be reduced by 0.05 (enhancing the weight of temporal features can supplement occult bone metabolism abnormalities not captured by imaging features).

[0023] Furthermore, the multi-task loss function formula for the dual-task training submodule is as follows:

[0024] in, Labels for actual fracture occurrence. To predict the probability of fracture risk; These are the actual time-series changes in bone metabolism indicators. This is a predictor of bone metabolism trends; Let cross-entropy be the loss function. Let be the mean squared error loss function.

[0025] The multi-task loss function is set with a weight of 0.7 for the main task and 0.3 for the sub-task. This ensures the priority of the core objective of "fracture risk prediction" while the sub-task of "bone metabolism trend prediction" helps to learn the correlation between bone metabolism and fracture. In small sample scenarios, this effectively avoids model overfitting. At the same time, the sub-task can help verify the logic of the main task (such as the positive correlation between the trend of bone metabolism deterioration and fracture risk), improve the clinical rationality of the model's prediction results, and avoid predictions that are detached from pathological logic due to pure data-driven approaches.

[0026] Furthermore, the interpretability module also includes a feature contribution visualization submodule; the feature contribution visualization submodule is used to display the contribution of each dynamic feature in the form of a waterfall chart.

[0027] The feature contribution visualization submodule visually displays the contribution of each feature in the form of a waterfall chart (e.g., "Osteolytic lesion enlargement contributes 35%, elevated blood calcium contributes 20%, and increased pain score contributes 15%"). Doctors can quickly locate core risk factors without having to analyze raw data one by one, effectively shortening the clinical decision-making time. At the same time, the visualization results can be used for patient communication (e.g., explaining to patients that "the main reason for the increased risk is the enlargement of the lesion").

[0028] Furthermore, the risk levels include three levels: high risk, medium risk, and low risk. High risk corresponds to a fracture risk probability of ≥65%, medium risk corresponds to a fracture risk probability of ≥30% and <65%, and low risk corresponds to a fracture risk probability of <30%. In case of high risk, the warning information and the text report are marked in red; in case of medium risk, the warning information and the text report are marked in yellow; and in case of low risk, the warning information and the text report are marked in green.

[0029] The warning information and text reports are marked with different colors according to different risk levels, so as to realize the hierarchical classification of information pushed to the user terminal and make it easier for users to quickly identify the current status. Attached Figure Description

[0030] Figure 1 This is a system module diagram of an embodiment of the deep learning-based fracture risk warning system for multiple myeloma patients according to the present invention. Detailed Implementation

[0031] The following detailed description illustrates the specific implementation method: The basic implementation examples are as follows: Figure 1 As shown: A fracture risk early warning system for multiple myeloma patients based on deep learning, including a data acquisition module, a multimodal fusion module, a prediction model module, an interpretability module, and an early warning push module; The data acquisition module is used to collect dynamic multimodal data from patients with multiple myeloma. This dynamic multimodal data includes dynamic clinical data, dynamic imaging data, and dynamic laboratory data. The dynamic clinical data includes disease staging, pain scores, and records of bone-targeted drug use. The dynamic imaging data includes three-dimensional CT images of the skeleton and MRI images of bone marrow edema. The dynamic laboratory data includes serum calcium and serum... - Microglobulins, alkaline phosphatase; A multimodal fusion module is used for feature extraction and fusion of dynamic multimodal data. This module includes an image feature extraction submodule, a temporal feature extraction submodule, and a feature fusion submodule. The image feature extraction submodule processes 3D CT images to extract features related to osteolytic lesions and processes MRI images to extract features related to bone marrow edema, outputting an image feature vector. The temporal feature extraction submodule processes dynamic clinical data and dynamic laboratory data, capturing trend changes in indicators and outputting a temporal feature vector. The feature fusion submodule maps the image feature vector and the temporal feature vector to the same dimension using a fully connected layer, determines the association weights of the two types of features using a pre-defined multiple myeloma bone disease pathology association rule base, and performs feature fusion using the following formula, outputting a multimodal fused feature vector:

[0032] in, This is a multimodal fusion feature vector. For image feature vectors, For time series feature vectors, For association weights; A prediction model module is used to predict fracture risk based on multimodal fusion feature vectors. The prediction model module includes a Transformer prediction model and a dual-task training submodule. The Transformer prediction model takes the multimodal fusion feature vectors as input and outputs the probability of fracture risk for the patient within a preset time period. The dual-task training submodule takes fracture risk prediction as the main task and bone metabolism deterioration trend prediction as the secondary task. It optimizes the Transformer prediction model through a multi-task loss function to improve the model's generalization ability and prediction accuracy for fracture risk prediction. The multi-task loss function is obtained by weighted averaging of the cross-entropy loss function of the main task and the mean squared error loss function of the secondary task. An interpretability module is used to parse the feature contribution logic of the fracture risk probability. The feature contribution logic is constructed based on the dynamic features corresponding to the dynamic clinical data, dynamic imaging data, and dynamic laboratory data. The interpretability module includes a SHAP calculation submodule and an association reporting submodule. The SHAP calculation submodule is used to calculate the contribution of each dynamic feature to the fracture risk probability. The association reporting submodule is used to generate a text report on the correlation between the changes of each dynamic feature and the fracture risk. The correlation is determined based on the pathological association rule library of multiple myeloma bone disease. The early warning push module is used to generate early warning information based on the fracture risk probability. The early warning information includes the risk level, the risk location, and intervention suggestions. The early warning information and the text report are pushed to the user terminal. The higher the fracture risk probability, the higher the risk level.

[0033] The data acquisition module also includes a data preprocessing submodule; the data preprocessing submodule performs normalization processing on the three-dimensional CT images and smoothing processing on the dynamic time series data to remove outlier interference.

[0034] The dynamic clinical data and dynamic laboratory data are updated according to a first fixed cycle, and the dynamic imaging data are collected according to a second fixed cycle. The first fixed cycle and the second fixed cycle are adapted to the clinical follow-up pattern of multiple myeloma patients. The three-dimensional CT images adopt the bone window reconstruction mode, and the bone marrow edema sequence MRI images adopt T2 fat-suppressed images. The disease staging is the ISS staging, and the pain score is the VAS score.

[0035] The association weight determination rule for the feature fusion submodule is as follows: when there is a pathological association between the image feature vector and the time-series feature vector, the association weight is assigned a value greater than 0.5; when there is no pathological association between the image feature vector and the time-series feature vector, the association weight is assigned a value of 0.5.

[0036] The feature fusion submodule also includes an attention assistance unit; the attention assistance unit fine-tunes the association weights based on the patient's individual medical history, with the fine-tuning range not exceeding 0.1.

[0037] The multi-task loss function formula for the dual-task training submodule is as follows:

[0038] in, Labels for actual fracture occurrence. To predict the probability of fracture risk; These are the actual time-series changes in bone metabolism indicators. This is a predictor of bone metabolism trends; Let cross-entropy be the loss function. Let be the mean squared error loss function.

[0039] The interpretability module also includes a feature contribution visualization submodule; the feature contribution visualization submodule is used to display the contribution of each dynamic feature in the form of a waterfall chart.

[0040] The risk levels are categorized into three levels: high risk, medium risk, and low risk. High risk corresponds to a fracture risk probability of ≥65%, medium risk corresponds to a fracture risk probability of ≥30% and <65%, and low risk corresponds to a fracture risk probability of <30%. For high risk, the warning message and text report are highlighted in red; for medium risk, they are highlighted in yellow; and for low risk, they are highlighted in green. By using different colors to highlight the warning message and text report according to different risk levels, the information pushed to the user's terminal is categorized, allowing the user to quickly identify the current status.

[0041] In this embodiment, the specific implementation process is as follows: 1. Dataset Construction and Preprocessing: The dataset was derived from a cohort of 800 MM patients, including 420 males (52.5%) and 380 females (47.5%), aged 45-78 years (mean 62.3 years). ISS staging included 180 cases of stage I, 420 cases of stage II, and 200 cases of stage III, with 240 patients (30%) suffering from fractures. Fracture sites included: spine (96 cases), femur (60 cases), ribs (48 cases), and humerus (36 cases). Each patient's data included at least baseline and two follow-up observations. The first fixed period was 2 weeks, and the second fixed period was set at 3 months to balance radiation dose and monitor bone structure changes. Imaging data covered key areas of the spine, ribs, femur, and humerus.

[0042] Data grouping: The data was divided into a training set (560 cases, including 168 fracture patients), a validation set (160 cases, including 48 fracture patients), and a test set (80 cases, including 24 fracture patients) in a 7:2:1 ratio.

[0043] Preprocessing: Image data: CT images were annotated with 3D Slicer to mark the vertebral body, femur and other bones and lesion ROIs in each location, and normalized after resampling; MRI T2 fat-suppressed images were segmented with U-Net to divide the bone marrow edema areas in each location and the edema area percentage was calculated; Time series data: Time series data such as pain VAS score and blood calcium were smoothed with a sliding window (window size 3) to remove outliers.

[0044] 2. Software environment: The deep learning framework used is PyTorch 2.0, medical image processing uses MONAI 1.2 (adapted for multi-site image segmentation and feature extraction), interpretability computation uses SHAP 0.42.1, and visualization uses Matplotlib 3.7 and Plotly 5.15 to generate waterfall plots and multi-site risk heatmaps.

[0045] 3. Model Training and Optimization Training parameters: The optimizer is AdamW, the learning rate is 5e-5, and the weight decay is 1e-4; 10-fold cross-validation is used, with an early stopping strategy (training stops if the validation set accuracy does not improve for 3 consecutive rounds); the main task label is "whether any fracture has occurred in any part of the body within 6 months" (including location annotation), and the sub-task label is "monthly changes in blood calcium and alkaline phosphatase"; the loss function weights are set at 0.7:0.3, and the main task and sub-task parameters are updated synchronously through backpropagation; The feature fusion submodule maps image feature vectors and temporal feature vectors to the same 64 dimensions through a fully connected layer, and outputs a 64-dimensional multimodal fusion feature vector after feature fusion. The Transformer prediction model employs a shared encoder and a dual-output branch structure, simultaneously supporting both the main and secondary tasks. Its specific construction is as follows: The encoder part contains a 6-layer encoder with a consistent structure. Each layer consists of a multi-head self-attention layer, a layer normalization layer, a feedforward network layer, and another layer normalization layer. The multi-head self-attention layer has 8 attention heads, each with a dimension of 64, resulting in a total hidden layer dimension of 512. It uses a scaled dot product attention mechanism, with the input being a multimodal fusion feature vector (64 dimensions), which is linearly projected to generate the query (Q) and key (K). Value (V) vectors (all with dimensions 512), the attention output is linearly transformed and added to the residual connection before entering the layer normalization; Feedforward network layer: contains two fully connected layers, the first layer has a hidden dimension of 2048 and uses ReLU activation function, the second layer has an output dimension of 512, realizing non-linear feature mapping and dimension restoration; Layer normalization layer: the normalization parameter is set to 1e-6 to prevent numerical overflow; Dropout configuration: dropout layers are added after the multi-head self-attention layer and the feedforward network layer, with a dropout rate of 0.1 to alleviate overfitting; Output Branches (Separation of Main and Sub-Tasks): Main Task Branch (Fracture Risk Prediction): The encoder output (512 dimensions) is mapped to probability values ​​through two fully connected layers. The first layer has a hidden dimension of 512 and uses ReLU as the activation function. The second layer has an output dimension of 1 and uses Sigmoid as the activation function. The final output is the fracture risk probability in the 0-1 range. Sub-Task Branch (Bone Metabolism Deterioration Trend Prediction): The 512-dimensional features output by the encoder are reused. The output is processed through a fully connected layer (256 hidden dimensions and ReLU activation function) and an output layer (2 output dimensions, corresponding to the monthly change prediction values ​​of blood calcium and alkaline phosphatase). There is no activation function. The continuous trend prediction result is directly output.

[0046] The dual-task training submodule jointly optimizes the Transformer prediction model through a multi-task loss function, as follows: Feature sharing: The main task and the sub-task share the encoder part. The multimodal fusion features learned by the encoder serve both types of tasks simultaneously, avoiding feature redundancy and improving data utilization; Task adaptation: The separate output branches are matched to the characteristics of the tasks respectively, where the main task is a binary classification problem and the sub-task is a continuous value regression problem; Joint optimization: The multi-task loss function is obtained by weighted averaging of the cross-entropy loss of the main task and the mean squared error loss of the sub-task. During backpropagation, the gradient of the loss function synchronously updates the parameters of the encoder and the two output branches, so that the features learned by the encoder not only meet the classification requirements of fracture risk prediction, but also conform to the regression law of bone metabolism trend. The additional supervision signal provided by the sub-task alleviates the overfitting caused by the scarcity of fracture samples in the main task, ultimately improving the generalization ability and prediction accuracy of the main task.

[0047] Weighting rules: A pathological association rule base is constructed based on the "Chinese Guidelines for the Diagnosis and Treatment of Multiple Myeloma" (2024 revision). For example, "osteolytic lesions of the femur" and "elevated serum calcium," and "rib edema" and "elevated alkaline phosphatase" are considered associated, with corresponding weights. Assign values ​​of 0.7 and 0.6 respectively; "spinal lesions" and "weight changes", "humeral lesions" and "serum" "Normal fluctuations in microglobulins" were deemed unrelated; correlation weight... Assign a value of 0.5.

[0048] The attention-assisted unit fine-tunes the correlation weights based on factors such as "history of fracture" and "duration of denosumab use". The magnitude should not exceed 0.1. For example, patients with a history of fractures have poorer bone structural stability, and the association weight can be increased by 0.05 (enhancing the dominant role of imaging features in risk prediction); another example is that patients using denosumab (a bone-targeted drug) long-term have more stable bone metabolism, and the effectiveness of trend changes in temporal features (such as serum calcium and alkaline phosphatase) in indicating fracture risk is reduced, so the association weight can be increased by 0.05 (enhancing the weight of imaging features and balancing the impact of decreased effectiveness of temporal features); yet another example is the serum... - The temporal changes in microglobulin not only reflect the condition of MM, but are also directly related to the interference of renal function on bone metabolism (the risk of occult bone resorption). At this time, the effectiveness of temporal features in indicating fracture risk is significantly improved, and the correlation weight can be reduced by 0.05 (enhancing the weight of temporal features can supplement occult bone metabolism abnormalities not captured by imaging features).

[0049] High risk corresponds to a fracture risk probability greater than or equal to 65%, medium risk corresponds to a fracture risk probability greater than or equal to 30% and less than 65%, and low risk corresponds to a fracture risk probability less than 30%. The preset time period is 6 months, that is, the fracture risk probability of the patient within 6 months is output by the Transformer prediction model.

[0050] The following will further illustrate this with specific scenarios: Scenario 1: Intervention for a high-risk femoral patient. The patient is a 65-year-old male, diagnosed with MM 2 years ago, ISS stage III. Follow-up data shows that "the osteolytic lesion in the left femur decreased from 0.8 cm to 0.8 cm within 3 months." 3 Increased to 1.5cm 3 "Serum calcium increased from 2.1 mmol / L to 2.4 mmol / L, and the VAS pain score (femoral region) increased from 3 to 6." According to the pathological association rule library for multiple myeloma bone disease, "osteolytic lesions of the femur (imaging features)" and "elevated serum calcium and increased femoral pain score (temporal features)" have a direct pathological association of "bone structure destruction, leading to abnormal bone metabolism, and thus exacerbating pain." The association weights were initialized. The value was 0.7; the system retrieved the patient's past medical history, which stated that the patient "had a left femoral fracture (non-pathological) 1 year ago, and recovered well after internal fixation surgery."

[0051] The attention-assisted unit fine-tunes the weights based on this: it adjusts the associated weights. Increase by 0.05, adjusted association weight The value is 0.75. After feature fusion in the multimodal fusion module, the resulting multimodal fusion feature vector is input into the Transformer prediction model, which outputs a femoral fracture risk probability of 72% (high risk) in the next 6 months.

[0052] The SHAP calculation submodule calculates the contribution of each dynamic feature to the fracture risk probability, ranked as follows: increased left femoral osteolytic lesion size (30%), elevated serum calcium (22%), increased VAS score for left femoral pain (18%), history of fracture (12%), duration of denosumab use (8%), and other features (10%). The feature contribution visualization submodule generates a feature contribution waterfall chart to visually display the risk composition. The association report submodule generates a text report based on the pathological association rule library for multiple myeloma bone disease, analyzing the pathological association between each dynamic feature and fracture risk. For example, the increased left femoral osteolytic lesion size (contribution 30%, primary risk factor) is pathologically associated with the left femoral osteolytic lesion volume decreasing from 0.8 cm within 3 months. 3 Increased to 1.5cm 3Significantly elevated bone density heterogeneity in the lesion area directly leads to destruction of trabecular bone structure and decreased bone load-bearing capacity (a direct manifestation of the core pathological mechanism of MM bone disease, where bone resorption exceeds bone formation), reducing the left femur's resistance to external impact and serving as the pathological basis for increased fracture risk. Elevated serum calcium (contributing 22%, a minor risk factor) is pathologically associated with a rise in serum calcium levels from 2.1 mmol / L to 2.4 mmol / L (exceeding the upper limit of the normal range), indicating increased MM-related bone resorption (enhanced osteoclast activity, leading to increased bone calcium release), further exacerbating bone matrix loss and forming a cycle with osteolytic lesions: "bone structure destruction, leading to bone calcium release, which in turn leads to further decline in bone strength."

[0053] The early warning push module generates early warning information based on the fracture risk probability, including fracture risk probability (72%), risk level (high risk), risk location (left femur), and intervention recommendations (bone cement reinforcement and denosumab dosage adjustment). The early warning information and text report are highlighted in red and pushed to the attending physician's clinical workstation terminal and the patient's electronic health record system. It is recommended to complete a three-dimensional CT angiography of the left femur within 72 hours to assess the femoral bone cement reinforcement femoral bone, with the surgical window not exceeding 7 days; adjust the denosumab dosage, as the current dose is insufficient to control elevated blood calcium (bone resorption remains active), and appropriately increase it. After adjustment, monitor blood calcium levels every 2 weeks, targeting 2.0–2.2 mmol / L to prevent further bone damage due to hypercalcemia; daily protection recommendations include limiting weight-bearing activities on the left femur (avoiding climbing stairs and lifting heavy objects), using assistive walking devices (such as a single crutch); and supplementing with calcium and vitamin D to help improve bone metabolism.

[0054] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A deep learning-based multiple myeloma patient fracture risk early warning system, characterized in that: The system comprises a data collection module, a multi-modal fusion module, a prediction model module, an explainability module and an early warning pushing module. A data acquisition module is configured to acquire dynamic multi-modal data of a multiple myeloma patient, wherein the dynamic multi-modal data comprises dynamic clinical data, dynamic imaging data and dynamic laboratory data; the dynamic clinical data comprises disease stage, pain score and bone-targeted drug usage record; the dynamic imaging data comprises three-dimensional CT image of the skeleton and bone marrow edema sequence MRI image; the dynamic laboratory data comprises blood calcium, serum microglobulin, alkaline phosphatase; The multi-modal fusion module is configured to extract and fuse features of dynamic multi-modal data, and comprises an image feature extraction submodule, a time series feature extraction submodule and a feature fusion submodule. The image feature extraction submodule is configured to process three-dimensional CT images to extract features related to osteolytic lesions, process MRI images to extract features related to bone marrow edema, and output an image feature vector. The time series feature extraction submodule is configured to process dynamic clinical data and dynamic laboratory data, capture trend change features of indexes, and output a time series feature vector. The feature fusion submodule is configured to map the image feature vector and the time series feature vector to the same dimension through a full connection layer, determine the correlation weight of the two types of features through a pre-set multiple myeloma bone disease pathology correlation rule base, perform feature fusion through the following formula, and output a multi-modal fusion feature vector: wherein, is a multi-modal fusion feature vector, is an image feature vector, is a time-series feature vector, is a correlation weight; The prediction model module is configured to predict bone fracture risk based on the multi-modal fusion feature vector. The prediction model module comprises a Transformer prediction model and a double-task training submodule. The Transformer prediction model takes the multi-modal fusion feature vector as input and outputs the probability of bone fracture risk of a patient in a pre-set time period in the future. The double-task training submodule takes bone fracture risk prediction as the main task and bone metabolism deterioration trend prediction as the auxiliary task. The Transformer prediction model is optimized through a multi-task loss function to improve the generalization ability and prediction accuracy of the model for bone fracture risk prediction. The multi-task loss function is obtained by weighted average of the cross-entropy loss function of the main task and the mean square error loss function of the auxiliary task. The explainability module is configured to analyze the feature contribution logic of the bone fracture risk probability. The feature contribution logic is constructed based on the dynamic features corresponding to the dynamic clinical data, dynamic image data and dynamic laboratory data. The explainability module comprises a SHAP calculation submodule and an association report submodule. The SHAP calculation submodule is configured to calculate the contribution degree of each dynamic feature to the bone fracture risk probability. The association report submodule is configured to generate a text report of the association between the change of each dynamic feature and bone fracture risk. The association is determined based on the multiple myeloma bone disease pathology correlation rule base. The early warning pushing module is configured to generate early warning information according to the bone fracture risk probability. The early warning information includes risk level, risk site and intervention suggestion. The early warning information and the text report are pushed to a user terminal. The greater the bone fracture risk probability, the higher the risk level.

2. The deep learning-based multiple myeloma patient bone fracture risk early warning system of claim 1, wherein: The data collection module further comprises a data preprocessing submodule. The data preprocessing submodule performs normalization processing on three-dimensional CT images and smoothing processing on dynamic time series data to remove abnormal value interference.

3. The deep learning-based multiple myeloma patient bone fracture risk early warning system of claim 1, wherein: The dynamic clinical data and dynamic laboratory data are updated at a first fixed cycle, and the dynamic image data are collected at a second fixed cycle, the first fixed cycle and the second fixed cycle are adapted to the clinical follow-up rule of the multiple myeloma patient; the three-dimensional CT image adopts a bone window reconstruction mode, and the bone marrow edema sequence MRI image adopts a T2 fat-suppressed image; the disease stage is an ISS stage, and the pain score is a VAS score.

4. The deep learning-based multiple myeloma patient bone fracture risk early warning system of claim 1, wherein: The association weight determination rule of the feature fusion sub-module is that when the image feature vector and the time series feature vector have pathological association, the association weight assignment is greater than 0.5; when the image feature vector and the time series feature vector do not have pathological association, the association weight assignment is 0.

5.

5. The deep learning-based multiple myeloma patient bone fracture risk early warning system of claim 1, wherein: The feature fusion sub-module further comprises an attention auxiliary unit. The attention auxiliary unit fine-tunes the association weight based on the individual medical history of the patient, and the fine-tuning amplitude is not more than 0.

1.

6. The deep learning-based multiple myeloma patient bone fracture risk early warning system of claim 1, wherein: The multi-task loss function formula of the double-task training sub-module is: wherein, is a label for actual occurrence of fracture, is a fracture risk prediction probability; is an actual time-series change value of a bone metabolism index, is a bone metabolism trend prediction value; is a cross-entropy loss function, is a mean square error loss function.

7. The deep learning-based multiple myeloma patient bone fracture risk early warning system of claim 1, wherein: The explainability module further comprises a feature contribution visualization sub-module. The feature contribution visualization sub-module is used to display the contribution degree of each dynamic feature in the form of a waterfall chart.

8. The deep learning-based multiple myeloma patient bone fracture risk early warning system of claim 1, wherein: The risk level includes three levels of high risk, medium risk and low risk, wherein the high risk corresponds to a fracture risk probability greater than or equal to 65%, the medium risk corresponds to a fracture risk probability greater than or equal to 30% and less than 65%, and the low risk corresponds to a fracture risk probability less than 30%, the pre-warning information and the text report are marked in red when the risk is high, the pre-warning information and the text report are marked in yellow when the risk is medium, and the pre-warning information and the text report are marked in green when the risk is low.

Images