Hip fracture surgical decision support system based on multi-modal artificial intelligence

Abstract

The invention provides a hip fracture operation type decision support system based on multi-modal artificial intelligence, and relates to the technical field of artificial intelligence medical assistance, and the system comprises a data obtaining module which is used for obtaining a preoperative pelvic CT image and electronic medical record structured information of a hip fracture patient; the feature extraction module is used for extracting features of a fracture key part in the preoperative pelvic CT image and encoding electronic medical record structured information into clinical features; the fusion prediction module is used for performing cross-modal fusion on the fracture key part features and the clinical features to obtain multi-modal fusion features, and inputting the multi-modal fusion features into a prediction model obtained by pre-training to obtain the risk probability that the hip fracture patient has clinical failure and needs secondary joint replacement. Therefore, clinical doctors can be assisted to make preoperative decision-making of hip fracture. The method has the beneficial effects that through cross-modal feature fusion, the objectivity, the stability and the clinical availability of an evaluation result are remarkably improved, and stable and reliable preoperative decision-making assistance can be provided for clinicians.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-assisted medical technology, and in particular to a decision support system for hip fracture surgery based on multimodal artificial intelligence. Background Technology

[0002] Hip fractures are a common type of fracture in clinical practice, affecting people from young adults to the elderly. Intracapsular hip fractures are more common in young and middle-aged patients. These fractures not only carry a high risk of disability but also require substantial long-term postoperative care, placing a heavy burden on patients' families and the social healthcare system. In clinical practice, for young and middle-aged patients with intracapsular hip fractures, clinicians often prioritize hip-preserving internal fixation surgery. The core objective is to preserve the patient's joint function, improve their postoperative quality of life, and reduce the trauma and subsequent complications associated with joint replacement surgery.

[0003] However, hip-preserving treatment still faces stubborn challenges: clinical practice data shows that the clinical failure rate after hip-preserving treatment can be as high as about 40%. Among patients who experience clinical failure, a considerable number eventually need to undergo a second joint replacement surgery. This not only increases the patient's pain and medical expenses, but also reduces the patient's satisfaction with the treatment results, and brings many troubles to clinical diagnosis and treatment.

[0004] To address the aforementioned issues and achieve accurate prognosis assessment and rational surgical selection for patients with hip fractures, current clinical practice largely relies on preoperative X-ray assessment of fracture displacement (e.g., Garden classification). Completely displaced fractures (Garden types 3 and 4) have a poor prognosis. For patients over 60 years of age, direct joint replacement is often recommended to reduce the risk of postoperative clinical failure. While the aforementioned assessment method based on preoperative X-ray classification is widely used in clinical practice, it still has significant limitations, mainly in the following aspects: First, two-dimensional X-rays are easily affected by factors such as patient position, limb rotation, structural overlap, and imaging quality, making it difficult to accurately reflect the true three-dimensional morphology of the fracture and to precisely capture the key morphological information of the fracture, which is precisely the important basis for assessing the severity of the fracture, judging the prognosis, and selecting the surgical procedure.

[0005] Second, the classification interpretation is highly subjective and has limited repeatability, which may lead to differences in conclusions among different operators or under different conditions. It is difficult to form a unified and standardized evaluation standard, which in turn affects the consistency of diagnosis and treatment decisions.

[0006] Third, such methods often use the degree of displacement or classification conclusion as the main or even the only basis for predicting prognosis and guiding surgical procedure selection. They are difficult to fully characterize the details of fracture morphology and individual differences, thus the prediction accuracy is limited. They cannot achieve accurate assessment of the patient's clinical failure and the risk of secondary joint replacement, nor can they provide stable and reliable individualized surgical procedure decision support.

[0007] Therefore, there is an urgent need for a more scientific, precise, and targeted preoperative assessment technology to assist clinicians in optimizing treatment decisions, reducing the clinical failure rate of hip-preserving treatment, decreasing the incidence of secondary joint replacement surgery, and improving patient prognosis. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a hip fracture surgical decision support system based on multimodal artificial intelligence, comprising: The data acquisition module is used to acquire preoperative pelvic CT images and structured information from electronic medical records of patients with hip fractures. The feature extraction module, connected to the data acquisition module, is used to extract the key fracture features in the preoperative pelvic CT images and encode the structured information of the electronic medical record into clinical features. The fusion prediction module, connected to the feature extraction module, is used to perform cross-modal fusion of the features of the key fracture sites and the clinical features to obtain multimodal fusion features. Then, the multimodal fusion features are input into a pre-trained prediction model to predict the risk probability of clinical failure and the need for secondary joint replacement in patients with hip fractures, so as to assist clinicians in making preoperative surgical decisions for hip fractures.

[0009] Preferably, the feature extraction module includes: The image segmentation unit is used to segment the hip fracture area from the preoperative pelvic CT image. An extraction unit, connected to the image segmentation unit, is used to identify key fracture sites in the hip fracture region, then determine the fracture ends from the key fracture sites, and then generate a three-dimensional region of interest based on the fracture ends expanding outward within the range of the key fracture sites as the features of the key fracture sites.

[0010] Preferably, the image segmentation unit includes: The coarse segmentation subunit is used to segment the anatomical entity of the hip bone within the pelvic region from the preoperative pelvic CT image. The fine segmentation subunit, connected to the coarse segmentation subunit, is used to segment the hip fracture region related to the fracture from the anatomical entity of the hip bone.

[0011] Preferably, the feature extraction module includes: The encoding unit is used to sequentially process the structured information of the electronic medical record for missing data processing, categorical variable encoding, and numerical variable standardization to obtain the clinical features.

[0012] Preferably, the fusion prediction module includes a multimodal self-attention fusion unit, comprising: The first transformation subunit is used to perform a linear transformation on the clinical features to obtain a query feature vector; The second transformation subunit is used to perform linear transformations on the key features of the fracture site and the corresponding global background features of the preoperative pelvic CT image to obtain the corresponding key feature vector and value feature vector. The weight calculation subunit is connected to the first transformation subunit and the second transformation subunit, and is used to calculate the normalization exponent of the query feature vector and the key feature vector to obtain the attention weight. The fusion subunit, which is connected to the second transformation subunit and the weight calculation subunit, is used to perform a weighted summation of the attention weights and the features of each dimension of the value feature vector to obtain the image-side weighted features, and to perform feature concatenation and fusion of the image-side weighted features and the query feature vector to obtain the multimodal fusion features.

[0013] Preferably, it further includes a first interpretable result output module, connected to the fusion prediction module, for generating an attention heatmap based on the attention weights, and overlaying the attention heatmap onto the preoperative pelvic CT image and outputting it.

[0014] Preferably, it also includes a second interpretable result output module, connected to the fusion prediction module, for obtaining and outputting the feature alignment relationship between the clinical features and the features of the key fracture sites through adaptive learning of the attention weights.

[0015] Preferably, it further includes a training module connected to the fusion prediction module, the training module comprising: The model building unit is used to build the prediction model, which includes a main task branch, an auxiliary task branch, and a joint loss function. The input and output of the main task branch are the multimodal fusion feature and the risk probability, respectively, and the input and output of the auxiliary task branch are the multimodal fusion feature and the auxiliary phenotypic information, respectively. The model training unit, connected to the model building unit, is used to jointly train the main task branch and the auxiliary task branch using a joint loss function to obtain the prediction model.

[0016] Preferably, the fusion prediction module further includes a risk stratification unit, which is used to output a high-risk stratification result and suggest joint replacement when the risk probability is not less than a preset first threshold, output a low-risk stratification result and suggest internal fixation when the risk probability is not greater than a second threshold, and output a medium-risk stratification result when the risk probability is greater than the second threshold and less than the first threshold, for clinicians to refer to when making preoperative surgical decisions.

[0017] Preferably, it also includes an attribution analysis module, connected to the fusion prediction module, used to calculate and output the contribution of the clinical features and the features of the key fracture sites to the risk probability, the key driving factors that have the most significant impact on the risk probability, and the direction of their impact on the risk probability, respectively, through feature attribution methods.

[0018] The above technical solution has the following advantages or beneficial effects: 1) This invention uses preoperative pelvic CT images and structured information from electronic medical records as joint inputs to achieve deep cross-modal fusion of key fracture features and clinical characteristics, fully characterizing fracture morphological details and individual patient differences. It effectively avoids the drawbacks of existing technologies that rely solely on two-dimensional X-ray classification to infer prognosis, and solves the core problems of inaccurate judgment and difficulty in reliably guiding surgical procedure selection. It can output more stable and quantifiable preoperative assessment results, significantly improving the objectivity, stability and clinical usability of the assessment results. 2) By fusing cross-modal features, it avoids the one-sided logic of using only the degree of shift or classification conclusion as the sole basis for assessment, and does not require manual interpretation by clinicians' experience, effectively reducing the influence of subjective factors and further improving the objectivity and consistency of assessment results. 3) It can provide clinicians with stable and reliable preoperative surgical decision-making assistance, especially suitable for young and middle-aged patients with intracapsular hip fractures. It helps doctors to scientifically judge the applicable scenarios of hip-preserving internal fixation and joint replacement surgery, effectively reducing clinical failures and secondary surgeries caused by improper initial surgical selection. This can reduce patients' repeated hospitalizations, revision surgeries and rehabilitation-related expenses, alleviate patients' pain and medical expenses, improve patients' long-term functional recovery, increase the possibility of patients' social reintegration, and achieve a dual improvement in diagnosis and treatment effects and medical benefits. Attached Figure Description

[0019] Figure 1 A schematic diagram of the structure of a hip fracture surgical decision support system based on multimodal artificial intelligence, which is a preferred embodiment of the present invention; Figure 2 In a preferred embodiment of the present invention, a schematic diagram of the auxiliary prediction process of the hip fracture surgical decision support system based on multimodal artificial intelligence is shown. Figure 3A schematic diagram of the cross-modal fusion process in a preferred embodiment of the present invention; Figure 4 This is a schematic diagram of the superposition effect of attention heatmaps in a preferred embodiment of the present invention. Figure 5 This is a schematic diagram of a decision curve in a preferred embodiment of the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The present invention is not limited to this embodiment; other embodiments that conform to the spirit of the present invention may also fall within the scope of the present invention.

[0021] In a preferred embodiment of the present invention, based on the above-mentioned problems existing in the prior art, a decision support system for hip fracture surgery based on multimodal artificial intelligence is provided. By integrating preoperative pelvic CT three-dimensional image information with electronic medical record structured information, the system can accurately predict the risk probability of clinical failure and the need for secondary joint replacement in patients with hip fractures. This provides stable and reliable auxiliary support for clinicians to make preoperative surgical decisions, and solves the drawbacks of the prior art that rely solely on two-dimensional X-ray classification assessment, which is inaccurate and difficult to guide surgical selection.

[0022] Based on actual clinical application scenarios, the hip fracture surgical decision support system of this invention, based on multimodal artificial intelligence, can be deployed on hospital servers or cloud computing nodes. It establishes connections with the hospital imaging system (PACS) and medical record system (HIS / EMR) through preset interfaces, forming a complete data loop without requiring additional specialized hardware, thus reducing clinical application costs. After deployment, the system automatically reads preoperative pelvic CT images from the PACS (all conforming to the DICOM medical digital imaging and communication standard) through the imaging interface, and automatically reads structured electronic medical record information from the HIS / EMR through the medical record interface. Simultaneously, using the patient's hospital number or visit number as a unique identifier, it maps the CT images and electronic medical record information of the same patient one-to-one at the patient level, ensuring data relevance and accuracy.

[0023] like Figure 1 As shown, the hip fracture surgical decision support system based on multimodal artificial intelligence of the present invention specifically includes a data acquisition module 1, a feature extraction module 2, a fusion prediction module 3, and may also optionally include a first interpretability result output module 4, a second interpretability result output module 5, a training module 6, and an attribution analysis module 7.

[0024] Specifically, the modules work together to achieve preoperative auxiliary prediction of hip fractures, and the specific auxiliary prediction process is as follows: Figure 2As shown, data acquisition module 1 acquires preoperative pelvic CT images of hip fracture patients in batches or individually through an interface with the hospital's PACS system. These pelvic CT images can be taken using standard clinical scanning protocols, and the slice thickness can be adjusted according to the hospital's equipment conditions (e.g., 1mm-3mm). No special scanning requirements are needed, as long as the fracture area can be accurately segmented and key fracture features can be extracted. Simultaneously, through an interface with the hospital's HIS / EMR system, the module acquires the patient's structured electronic medical record information. This structured information includes, but is not limited to: basic patient information (age, gender, BMI), comorbidity information (e.g., osteoporosis, diabetes, hypertension), lifestyle information (smoking and drinking history), and treatment-related information (admission to surgery time, past fracture history, etc.). All of this is routinely collected structured data clinically, requiring no additional data acquisition workload.

[0025] Preferably, the data acquisition module 1 also includes a data verification unit, which is used to verify the integrity of the acquired preoperative pelvic CT images, such as removing blurred or missing key areas of the images, and marking missing information in the structured information of the electronic medical record, such as marking missing variables, so that the subsequent feature processing module can process missing values ​​and ensure that the data input to the feature extraction module 2 meets the requirements of subsequent processing.

[0026] Feature extraction module 2, connected to data acquisition module 1, is used to extract the key fracture features from preoperative pelvic CT images and encode the structured information of electronic medical records into clinical features.

[0027] Furthermore, feature extraction module 2 includes: Image segmentation unit 21 is used to segment the hip fracture area from preoperative pelvic CT images; Extraction unit 22, connected to image segmentation unit 21, is used to identify key fracture sites in the hip fracture region, then determine the fracture ends in the key fracture sites, and then generate a three-dimensional region of interest based on the outward expansion of the fracture ends within the range of the key fracture sites as the features of the key fracture sites.

[0028] The image segmentation unit 21 precisely segments the hip fracture area from preoperative pelvic CT images, providing a precise regional range for subsequent extraction of key fracture features and avoiding interference from irrelevant areas of the entire image (such as soft tissues around the pelvis). In a preferred embodiment of the invention, the image segmentation unit 21 employs a two-step segmentation strategy of first coarse segmentation and then fine segmentation to improve segmentation accuracy. Specifically, the image segmentation unit 21 includes: The coarse segmentation subunit 211 is used to segment the anatomical entity of the hip bone in the pelvic region from the preoperative pelvic CT image. Preferably, it can use existing medical image segmentation algorithms and based on preset hip bone anatomical features to quickly segment the entire hip bone region (including key anatomical structures such as the femoral head, femoral neck, and proximal femur), remove irrelevant areas such as soft tissues and intestines around the pelvis, and obtain the preliminary segmentation result of the anatomical entity of the hip bone.

[0029] Fine segmentation subunit 212, connected to coarse segmentation subunit 211, is used to segment the hip fracture region related to the fracture from the anatomical entity of the hip bone.

[0030] In addition, the key fracture sites can be obtained either by the automatic identification algorithm in this embodiment or by using the region of interest (ROI) manually delineated by the clinician as input, as long as the core area for subsequent feature extraction can be clearly identified.

[0031] Based on the above segmentation, the extraction unit 22 analyzes the hip fracture area through the built-in fracture key site identification algorithm, identifies key sites such as fracture lines and fracture zones, and further determines the fracture ends (i.e. the fracture surface and fracture boundary between the proximal and distal bone fragments). These key sites are the core areas that reflect the severity of the fracture and predict the risk of clinical failure.

[0032] Subsequently, in order to highlight the morphological features of key areas and reduce interference from irrelevant information, the extraction unit 22 expands outwards at a preset distance within the critical fracture area to generate a three-dimensional region of interest (ROI). This expansion distance / size can use a fixed threshold (e.g., 5mm-10mm) or can be adaptively set according to the volume of the fracture area (the larger the volume of the fracture area, the larger the expansion distance should be to ensure coverage of all key morphological areas).

[0033] Finally, the extraction unit 22 takes the three-dimensional region of interest (ROI) as the core, extracts the fracture morphology features (such as fracture line direction, degree of bone fragment displacement, degree of comminutedness, etc.) within the region, and encodes them into a standardized image feature sequence as the key features of the fracture site, and outputs them to the fusion prediction module 3.

[0034] In a preferred embodiment of the present invention, the feature extraction module 2 further includes: Encoding unit 23 is used to sequentially process the structured information of electronic medical records by performing missing data processing, categorical variable encoding, and numerical variable standardization to obtain clinical features.

[0035] Specifically, in this embodiment, the missing value handling includes, but is not limited to, filling the numerical variables with the mean or median, and filling the categorical variables with the mode, to avoid missing values ​​interfering with subsequent analysis.

[0036] The above-mentioned categorical variable encoding includes, but is not limited to, converting categorical variables in electronic medical records (such as gender, type of comorbidity, smoking history, etc.) into numerical forms that can be used for calculation using conventional encoding methods such as one-hot encoding and label encoding.

[0037] The standardization of the above-mentioned numerical variables includes, but is not limited to, normalizing and standardizing the numerical variables in the electronic medical record (such as age, BMI, time from admission to surgery, etc.) to map the variable values ​​to the same numerical range (such as the 0-1 interval) to avoid interference from the difference in magnitude of the dependent variable on multimodal fusion.

[0038] After the above processing, the encoding unit 23 transforms the structured information of the electronic medical record into a standardized sequence of clinical features, which is then output as clinical features to the fusion prediction module 3, forming two types of multimodal features with the features of the key fracture sites. The fusion prediction module 3 is used to perform cross-modal fusion of the features of the key fracture sites and the clinical features to obtain multimodal fused features. These multimodal fused features are then input into a pre-trained prediction model to predict the probability of clinical failure and the need for a second joint replacement in patients with hip fractures, thereby assisting clinicians in making preoperative surgical decisions for hip fractures.

[0039] In a preferred embodiment of the present invention, the fusion prediction module 3 includes a multimodal self-attention fusion unit 31 for achieving cross-modal alignment and deep fusion of fracture key site features and clinical features. This overcomes the drawbacks of traditional simple feature stacking, and adaptively learns the correlation strength between the two types of features through an attention mechanism, strengthening the weight of high-risk features and improving the effectiveness of the fusion representation. Specifically, the multimodal self-attention fusion unit 31 includes: The first transformation subunit 311 is used to perform a linear transformation on the clinical features to obtain the query feature vector; The second transformation subunit 312 is used to perform linear transformations on the key features of the fracture site and the corresponding global background features of the preoperative pelvic CT image to obtain the corresponding key feature vector and value feature vector. The weight calculation subunit 313 is connected to the first transformation subunit 311 and the second transformation subunit 312 respectively, and is used to calculate the normalization exponent of the query feature vector and the key feature vector to obtain the attention weight. The fusion subunit 314 is connected to the second transformation subunit 312 and the weight calculation subunit 313 respectively. It is used to perform weighted summation of attention weights and features of each dimension of the value feature vector to obtain image-side weighted features, and to perform feature splicing and fusion of image-side weighted features and query feature vector to obtain multimodal fusion features.

[0040] Specifically, in this embodiment, as Figure 3As shown, the first transformation subunit 311 transforms clinical features into a query feature vector q with unified dimensions and adapted to attention calculation through a preset linear transformation matrix W_q, which serves as a retrieval benchmark for multimodal fusion and is used to retrieve image features related to clinical high-risk features.

[0041] The second transformation subunit 312 transforms it into a key feature vector k through a linear transformation matrix W_k, which serves as the matching benchmark for multimodal fusion; and transforms it into a value feature vector v through a linear transformation matrix W_v, which carries the specific details of the image side features.

[0042] Furthermore, the weight calculation subunit 313 performs a normalization exponent (Softmax) calculation on the query feature vector q and the key feature vector k to obtain the attention weight A (association strength coefficient matrix). Each element in the attention weight A corresponds to the degree of association between a set of clinical feature tokens and image feature tokens. The larger the value, the higher the degree of association between the corresponding image feature and clinical feature, and the higher the weight will be obtained in subsequent fusion.

[0043] Furthermore, the fusion subunit 314 first performs a weighted summation of the attention weight A and the features of each dimension of the value feature vector v to obtain the image-side weighted features after attention screening, so as to highlight the image features with high correlation with clinical high-risk features. Then, the image-side weighted features are concatenated and fused with the query feature vector q to obtain the multimodal fusion feature T_fused. This fusion feature simultaneously carries the fracture morphology information on the image side and the individual patient information on the clinical side, fully preserving the correlation between clinical high-risk features and unstable image morphology, and providing accurate feature support for subsequent risk prediction.

[0044] As can be seen, the multimodal self-attention fusion unit 31 generates high-resolution local tokens with the fracture line / fracture area and the three-dimensional region of interest (ROI) formed by its surrounding expansion as the core, so as to highlight subtle morphological features such as displacement, posterior cortical defects, and comminuted fragments; at the same time, it retains low-resolution global tokens covering the femoral head and neck and proximal femur to provide anatomical background and overall structural relationship, thereby taking into account key morphological information and global anatomical constraints, and further improving the effectiveness of fusion representation.

[0045] It is worth noting that when clinical features suggest high-risk characteristics such as advanced age, osteoporosis, or diabetes, the attention mechanism can automatically increase the image token weight corresponding to unstable imaging morphologies (such as posterior cortical defects, displacement or increased tilt angle, comminuted fragments, etc.), thereby strengthening the identification of unstable morphologies and improving the consistency and interpretability of preoperative risk assessment.

[0046] Among them, the aforementioned cross-modal fusion methods can adopt attention fusion, or can be replaced by conventional multimodal fusion methods in this field such as gating fusion or feature interaction, as long as they can reflect the effective interaction of the two types of information and achieve the synergistic association between clinical features and imaging features, rather than simply stacking them.

[0047] After processing and obtaining the aforementioned multimodal fusion features, inputting them into a pre-trained prediction model yields the probability of clinical failure and the need for secondary joint replacement in patients with hip fractures. A higher probability value indicates a higher risk of postoperative clinical failure and the need for secondary joint replacement. The prediction model preferably employs a multilayer perceptron (MLP), and improves its generalization ability and cross-center stability by introducing layer normalization and Dropout regularization, residual connections, and output probability calibration, thus avoiding overfitting.

[0048] In a preferred embodiment of the present invention, the fusion prediction module 3 further includes a risk stratification unit 32, which is used to output a high-risk stratification result and suggest joint replacement when the risk probability is not less than a preset first threshold, output a low-risk stratification result and suggest internal fixation when the risk probability is not greater than a second threshold, and output a medium-risk stratification result when the risk probability is greater than the second threshold and less than the first threshold, so as to provide reference for clinicians when making preoperative surgical decisions.

[0049] Specifically, in this embodiment, the first and second thresholds are determined based on training set data, preferably meeting the clinical requirements of PPV ≥ 0.90 (positive predictive value) and NPV ≥ 0.90 (negative predictive value). The first threshold is a high-risk threshold, and the second threshold is a low-risk threshold. When the risk probability is greater than or equal to the first threshold, a high-risk stratification result is output, indicating that the patient has an extremely high risk of clinical failure after surgery and the need for a second joint replacement. Clinicians are advised to prioritize primary joint replacement surgery. When the risk probability is less than or equal to the second threshold, the low-risk stratification result is output, indicating that the patient has a high probability of successful hip preservation after surgery, and it is recommended that clinicians give priority to one-stage hip preservation internal fixation surgery. When the risk probability is greater than the second threshold but less than the first threshold, the risk stratification result is output, indicating that the patient's risk is at an intermediate level. The clinician then makes the final surgical decision based on the patient's individual needs, clinical experience, and other clinical factors.

[0050] To further improve the robustness, stability, and interpretability of the prediction model, in a preferred embodiment of the present invention, a training module 6 is further included, connected to the fusion prediction module 3. The training module 6 includes: Model building unit 61 is used to build a prediction model that includes a main task branch, an auxiliary task branch, and a joint loss function. The input and output of the main task branch are multimodal fusion features and risk probability, respectively, and the input and output of the auxiliary task branch are multimodal fusion features and auxiliary phenotypic information, respectively. The model training unit 62, connected to the model building unit 61, is used to jointly train the main task branch and the auxiliary task branch using a joint loss function to obtain the prediction model.

[0051] Specifically, in this embodiment, the aforementioned auxiliary phenotypic information includes the degree of displacement, fracture location, and degree of comminutedness. The degree of displacement is defined quantitatively based on the Garden classification and combined with the anteversion / posterior tilt angle of the fracture. Preferably, Garden III–IV types, or Garden I–II types with an anteversion / posterior tilt angle >20°, are defined as "displaced"; Garden I–II types with an anteversion / posterior tilt angle <20° are defined as "non-displaced". The fracture location is preferably divided into subcephalic and non-subcephalic types. Subcephalic fractures refer to fractures where the main fracture line is confined to the predetermined I and IV zones, while non-subcephalic fractures refer to fractures where the main fracture line extends beyond the predetermined I and IV zones. The degree of comminutedness is preferably defined as the presence of independent bone fragments on CT images with any dimension >1 cm that are not continuous with the proximal or posterior bone fragments. Further grading methods or quantitative indicators such as the number of fragments or the proportion of fragment volume can be used to characterize the severity of the comminutedness.

[0052] During the training phase of the prediction model, the auxiliary task branch and the main task branch share the same image coding backbone and image-side feature tokens. The system sets up parallel output heads to predict three types of auxiliary phenotypic information: displacement degree, fracture location, and comminutedness. Together with the risk probability prediction of the main task, they form a multi-task joint learning framework. By setting a joint loss function, the auxiliary task is forced to share the multimodal fusion representation T_fused and image coding features with the main task. At the same time, it has the ability to represent key morphological factors of fractures in a distinguishable and interpretable manner. This plays a morphological regularization role in the risk probability prediction of the main task, effectively reducing the model overfitting and cross-center prediction instability caused by learning solely based on outcome labels. It significantly improves the representation ability of fracture three-dimensional morphology and the overall robustness of the model, laying the foundation for accurate inference of the model after training.

[0053] During the inference phase of the predictive model, auxiliary task branches can be selectively enabled or disabled according to clinical needs. It is preferable to set up a mode control unit to automatically control its start and stop according to the current operating mode of the model, such as automatically enabling the training mode and automatically disabling the inference mode.

[0054] Further optimized, when the auxiliary task branch is closed during the inference phase, the output of auxiliary phenotypic information is stopped, and the model only outputs the risk probability and risk stratification results, which can quickly complete the risk assessment and improve the inference efficiency. When there is a need for clinical review, confidence adjustment, or consistency correction, the auxiliary task branch can be selectively opened through the pattern control unit. At this time, the auxiliary phenotypic information output by the auxiliary task branch can be output synchronously with the risk value output by the main task, serving as morphological evidence of the risk prediction results of the main task. At the same time, it can be used as a conditional feature back-injected into the final prediction stage to perform consistency correction or confidence adjustment on the prediction results, thereby further improving the stability and interpretability of the main task output.

[0055] To enhance the transparency and clinical acceptability of system decisions and to assist in doctor-patient communication and clinical review, this system also includes a first interpretability output module 4 and a second interpretability output module 5, both of which are connected to the fusion prediction module 3. They output interpretable results from two dimensions: attention heatmap and feature alignment relationship, respectively.

[0056] The first interpretability result output module 4 is used to generate an attention heatmap based on attention weights, and then overlay and display the attention heatmap on the preoperative pelvic CT image and output it. The specific overlay location can be a three-dimensional region of interest (ROI) on the preoperative pelvic CT image or a fracture-related key section, such as... Figure 4 As shown, the model intuitively presents the fracture abnormal areas that it focuses on, allowing clinicians to clearly understand which fracture sites and morphological characteristics the model focuses on during risk prediction, providing intuitive evidence for clinical review.

[0057] The second interpretability output module 5 is used to obtain and output the feature alignment relationship between clinical features and key fracture site features through attention weight adaptive learning. This feature alignment relationship can clearly define the matching logic between single / multiple clinical feature tokens and corresponding imaging feature tokens. For example, the clinical feature of osteoporosis can correspond to the imaging features of abnormal bone density and easy bone displacement, allowing clinicians to clearly understand the decision logic of the model and improve the acceptability of the decision results.

[0058] In a preferred embodiment of the present invention, the invention further includes an attribution analysis module 7, which is connected to the fusion prediction module 3, and is used to calculate and output the contribution of clinical features and fracture key site features to the risk probability, the key driving factors that have the most significant impact on the risk probability, and the direction of their impact on the risk probability through feature attribution methods.

[0059] Specifically, in this embodiment, the attribution analysis module 7 uses conventional feature attribution techniques in the field (preferably SHAP attribution method) to quantitatively analyze the clinical features and fracture key site features input by the fusion prediction module 3, calculate the contribution of each type of feature to the individual preoperative risk prediction result, screen out the key driving factors that have the most significant impact on the risk probability, and clarify the direction of influence of each key driving factor on preoperative risk. This imaging direction includes: 1) positive influence: the presence of the factor / increased value increases the risk; 2) negative influence: the presence of the factor / increased value decreases the risk.

[0060] Furthermore, the quantitative analysis results output by the attribution analysis module 7 can work in conjunction with the content output by the interpretability results output module to provide intuitive and quantitative evidence for clinical review, thereby further improving the reliability and scientific nature of diagnosis and treatment decisions, assisting clinicians in more accurately judging the source of risk, and optimizing surgical procedures.

[0061] To verify the clinical applicability, predictive accuracy, and superiority of the system of this invention, clinical validation was conducted in a multi-center cohort. The specific validation results are as follows, demonstrating the technical effectiveness and clinical application value of the system: 1. Validation dataset: A total of 1125 patients with hip fractures were included, including a four-center development cohort of n=951 and a two-center external testing cohort of n=174. All patients were followed up for an average of 6.6 years postoperatively, and the incidence of secondary joint replacement was 21.2% (a total of 238 patients underwent secondary joint replacement). 2. Validation results of this system (MMHIP): In the development cohort, the AUROC (area under the curve, used to evaluate prediction accuracy) for predicting the risk of secondary joint replacement was 0.930, and the Brier score (used to evaluate prediction calibration performance) was 0.080; in the external testing cohort, the AUROC was 0.868, and the Brier score was 0.119, indicating that the system has high prediction accuracy and good calibration performance, and can meet the requirements of clinical application. 3. Multi-model comparison and verification: To demonstrate the advantages of the multi-modal fusion scheme of this invention, the system was compared with single-modal models and models with different modeling routes. The specific comparison results are as follows: (1)In the development cohort: For the ResNet model (only imaging modality), AUROC = 0.822, Brier = 0.170; for the ViT model (only imaging modality), AUROC = 0.802, Brier = 0.175; for the XGBoost model (only clinical modality), AUROC = 0.743, Brier = 0.154; for the random forest model (only clinical modality), AUROC = 0.706, Brier = 0.193; for the logistic regression model (only clinical modality), AUROC = 0.720, Brier = 0.178; (2)In the external test cohort: For the ResNet model, AUROC = 0.727, Brier = 0.265; for the ViT model, AUROC = 0.755, Brier = 0.230; for the XGBoost model, AUROC = 0.761, Brier = 0.136; for the random forest model, AUROC = 0.697, Brier = 0.200; for the logistic regression model, AUROC = 0.653, Brier = 0.178; The above control results show that the prediction accuracy (AUROC) and calibration performance (Brier score) of the system (MMHIP) of the present invention are significantly better than those of single - modality models, fully reflecting the technical advantages of multi - modality fusion and enabling more accurate risk prediction.

[0062] 4. Verification of clinical triage operability: Based on the risk stratification thresholds determined from the training set (n_1 = 0.758, n_2 = 0.707), verify the clinical triage effect of this system: (1)High - risk group (predicted value ≥ 0.758): In the development cohort, sensitivity = 0.577, PPV = 0.902 (TP = 120, FP = 13); in the external test cohort, sensitivity = 0.567, PPV = 0.895 (TP = 17, FP = 2), which can accurately identify high - risk patients and provide a reliable basis for recommending joint replacement surgical methods; (2)Low - risk group (predicted value ≤ 0.707): In the development cohort, specificity = 0.973, NPV = 0.900 (TN = 723, FN = 80); in the external test cohort, specificity = 0.924, NPV = 0.930 (TN = 133, FN = 10), which can accurately identify low - risk patients and provide a reliable basis for recommending hip - preserving internal fixation surgical methods; (3)Medium - risk group (n_2 < predicted value < n_1): Clinicians make a comprehensive decision in combination with patient needs, taking into account clinical flexibility; Under the same rule-in / rule-out framework, the control models showed lower overall sensitivity in the rule-in phase (0.029–0.240) and a significant decrease in external test PPV (0.467–0.647). Some EHR-based control models failed to meet the clinical requirement of PPV ≥ 0.90. Although similar NPVs could be achieved in the rule-out phase, the specificity ranged only from 0.497 to 0.766, suggesting a greater likelihood of unnecessary "non-low-risk" triages, further demonstrating the superior clinical applicability of our system.

[0063] 5. Decision Curve Analysis (DCA): such as Figure 5 The decision curve analysis shown indicates that, within each threshold probability range, the system of this invention has a higher net benefit than the extreme strategies of total joint replacement or total hip-preserving internal fixation and the single-modal control model, demonstrating its practical value in real clinical decision-making. It can effectively optimize clinical surgical decisions and improve treatment outcomes.

[0064] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made using the content of this specification and illustrations should be included within the protection scope of the present invention.

Claims

1. A decision support system for hip fracture surgery based on multimodal artificial intelligence, characterized in that, include: The data acquisition module is used to acquire preoperative pelvic CT images and structured information from electronic medical records of patients with hip fractures. The feature extraction module, connected to the data acquisition module, is used to extract the key fracture features in the preoperative pelvic CT images and encode the structured information of the electronic medical record into clinical features. The fusion prediction module, connected to the feature extraction module, is used to perform cross-modal fusion of the features of the key fracture sites and the clinical features to obtain multimodal fusion features. Then, the multimodal fusion features are input into a pre-trained prediction model to predict the risk probability of clinical failure and the need for secondary joint replacement in patients with hip fractures, so as to assist clinicians in making preoperative surgical decisions for hip fractures.

2. The decision support system for hip fracture surgery according to claim 1, characterized in that, The feature extraction module includes: The image segmentation unit is used to segment the hip fracture area from the preoperative pelvic CT image. An extraction unit, connected to the image segmentation unit, is used to identify key fracture sites in the hip fracture region, then determine the fracture ends from the key fracture sites, and then generate a three-dimensional region of interest based on the fracture ends expanding outward within the range of the key fracture sites as the features of the key fracture sites.

3. The decision support system for hip fracture surgery according to claim 2, characterized in that, The image segmentation unit includes: The coarse segmentation subunit is used to segment the anatomical entity of the hip bone within the pelvic region from the preoperative pelvic CT image. The fine segmentation subunit, connected to the coarse segmentation subunit, is used to segment the hip fracture region related to the fracture from the anatomical entity of the hip bone.

4. The hip fracture surgical decision support system according to claim 1, characterized in that, The feature extraction module includes: The encoding unit is used to sequentially process the structured information of the electronic medical record for missing data processing, categorical variable encoding, and numerical variable standardization to obtain the clinical features.

5. The decision support system for hip fracture surgery according to claim 1, characterized in that, The fusion prediction module includes a multimodal self-attention fusion unit, comprising: The first transformation subunit is used to perform a linear transformation on the clinical features to obtain a query feature vector; The second transformation subunit is used to perform linear transformations on the key features of the fracture site and the corresponding global background features of the preoperative pelvic CT image to obtain the corresponding key feature vector and value feature vector. The weight calculation subunit is connected to the first transformation subunit and the second transformation subunit, and is used to calculate the normalization exponent of the query feature vector and the key feature vector to obtain the attention weight. The fusion subunit, which is connected to the second transformation subunit and the weight calculation subunit, is used to perform a weighted summation of the attention weights and the features of each dimension of the value feature vector to obtain the image-side weighted features, and to perform feature concatenation and fusion of the image-side weighted features and the query feature vector to obtain the multimodal fusion features.

6. The decision support system for hip fracture surgery according to claim 5, characterized in that, It also includes a first interpretable result output module, connected to the fusion prediction module, for generating an attention heatmap based on the attention weights, and overlaying the attention heatmap onto the preoperative pelvic CT image and outputting it.

7. The decision support system for hip fracture surgery according to claim 5, characterized in that, It also includes a second interpretable result output module, connected to the fusion prediction module, used to obtain and output the feature alignment relationship between the clinical features and the features of the key fracture sites through adaptive learning of the attention weights.

8. The decision support system for hip fracture surgery according to claim 1, characterized in that, It also includes a training module connected to the fusion prediction module, the training module comprising: The model building unit is used to build the prediction model, which includes a main task branch, an auxiliary task branch, and a joint loss function. The input and output of the main task branch are the multimodal fusion feature and the risk probability, respectively, and the input and output of the auxiliary task branch are the multimodal fusion feature and the auxiliary phenotypic information, respectively. The model training unit, connected to the model building unit, is used to jointly train the main task branch and the auxiliary task branch using a joint loss function to obtain the prediction model.

9. The hip fracture surgical decision support system according to claim 1, characterized in that, The fusion prediction module also includes a risk stratification unit, which is used to output a high-risk stratification result and suggest joint replacement when the risk probability is not less than a preset first threshold, output a low-risk stratification result and suggest internal fixation when the risk probability is not greater than a second threshold, and output a medium-risk stratification result when the risk probability is greater than the second threshold and less than the first threshold, so as to provide clinicians with reference when making preoperative surgical decisions.

10. The decision support system for hip fracture surgery according to claim 1, characterized in that, It also includes an attribution analysis module, which is connected to the fusion prediction module. This module is used to calculate and output the contribution of the clinical features and the key fracture site features to the risk probability, the key driving factors that have the most significant impact on the risk probability, and the direction of their impact on the risk probability, respectively, through feature attribution methods.

Images