Intelligent montella fracture typing method and device based on image-text multi-modal fusion
By employing an intelligent classification method that integrates text and image multimodal fusion, combined with deep interactive fusion and cost-sensitive learning, the limitations of single-modal approaches and data imbalance in Monteggia fracture diagnosis are addressed, achieving highly accurate and comprehensive diagnosis of Monteggia fractures.
Patent Information
- Application Number
- CN202511846698.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-03-17
AI Technical Summary
The diagnosis of Monteggia fractures suffers from monomodal limitations, fragmented clinical text information, and data category imbalance, leading to a high risk of missed diagnoses, especially in cases of radial head dislocation. Existing computer-aided diagnostic systems cannot effectively simulate the comprehensive diagnostic logic of physicians.
An intelligent classification method based on image-text multimodal fusion is adopted. By deeply interacting and fusing visual features of radiological images with clinical text descriptions in electronic medical records, and combining feature interaction fusion module and cost-sensitive learning strategy, hierarchical aggregated feature vectors are generated, and higher misclassification penalty weights are applied to rare classifications during model training.
It significantly improves the diagnostic accuracy of Monteggia fractures, especially the ability to identify difficult cases, reduces the risk of missed diagnosis, ensures a high level of diagnostic and identification capabilities for various subtypes, and achieves comprehensive and accurate coverage of the Monteggia fracture disease spectrum.
Smart Images
Figure CN121686479A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image classification technology, and more specifically, to a method and device for intelligent classification of Monteggia fractures based on image-text multimodal fusion. Background Technology
[0002] Monteggia fracture is a complex upper limb injury, typically characterized by a fracture of the ulna combined with radial head dislocation. In clinical practice, this injury particularly tests the diagnostic abilities of inexperienced physicians, posing a very high risk of missed diagnosis. The key to missed diagnosis is usually not the ulna fracture itself (which is relatively obvious on X-rays), but rather the overlooking of the potentially occult radial head dislocation. Delayed diagnosis of radial head dislocation can lead to a series of serious consequences, including severe impairment of elbow joint function, chronic pain, developmental deformities, and even make subsequent reconstructive surgery extremely difficult, imposing a heavy physical and financial burden on the patient.
[0003] Currently, the diagnosis of Monteggia fractures mainly relies on the interpretation of X-ray images by radiologists or orthopedic surgeons, but this diagnostic process faces the following key technical bottlenecks:
[0004] First, the diagnostic criteria suffer from a single-modal limitation. Traditional diagnosis relies entirely on image information, with physicians making judgments by observing bone arrangement, fracture line morphology, and auxiliary lines. However, this purely visual method is highly subjective, and its accuracy depends heavily on the physician's clinical experience. Even experienced physicians may misdiagnose atypical cases or those with subtle dislocations.
[0005] Second, the clinical context is fragmented. In actual diagnosis and treatment, clinical textual information such as the patient's medical history and the mechanism of injury is crucial for the diagnosis of Monteggia fractures. For example, a child's medical history of elbow injury accompanied by significant pain and limited range of motion, combined with an X-ray that appears to be only a simple ulnar fracture, should raise a high suspicion of radial head dislocation. However, most existing computer-aided diagnostic (CAD) systems are monomodal, capable of analyzing only pixel-level image data, completely ignoring this crucial textual information, and unable to simulate the comprehensive diagnostic thinking of an excellent physician.
[0006] Third, real-world data presents a challenge of class imbalance. Our self-collected and labeled clinical dataset of Monteggia fractures shows significant differences in the frequency of different subtypes, with common subtypes comprising the majority of the dataset and rare subtypes being scarce. If a standard deep learning model is used for training, the model will be biased towards predicting the common subtypes with their large sample size, resulting in severely inadequate diagnostic performance for rare subtypes. This is unacceptable in medical applications requiring high reliability.
[0007] In view of this, this application proposes an intelligent classification method and device for Monteggia fracture based on image-text multimodal fusion, which aims to overcome the limitations of single-modality, deeply integrate the visual features of radiographic images with the clinical text descriptions of electronic medical records, effectively model the interaction relationship between image and text information, and simulate the comprehensive diagnostic logic of advanced physicians; at the same time, it overcomes the problem of clinical data class imbalance at the model training level, ensuring high diagnostic accuracy for rare classifications. Summary of the Invention
[0008] The purpose of this application is to provide a method and device for intelligent classification of Monteggia fractures based on image-text multimodal fusion, which improves the accuracy and comprehensiveness of disease prediction and diagnosis through deep interactive fusion, hierarchical information aggregation and cost-sensitive learning strategies.
[0009] This application provides an intelligent classification method for Monteggia fractures based on image-text multimodal fusion, including the following steps:
[0010] S1. Extract the image data and text data from the input sample respectively, and generate an original image feature vector and an original text feature vector;
[0011] S2. Based on the original image feature vector and the original text feature vector, a deep fusion feature vector is generated through a feature interaction fusion module;
[0012] S3. The deep fusion feature vector, the original image feature vector, and the original text feature vector are concatenated to form a hierarchical aggregated feature vector;
[0013] S4. Input the hierarchical aggregated feature vector into a classifier, and during model training, use a cost-sensitive loss function to optimize the classifier, wherein the weight parameters of the loss function are predetermined based on the distribution of samples of each class in the training dataset.
[0014] Further, in step S1, the original image feature vector is extracted by a pre-trained ViT model, and the original text feature vector is extracted by a pre-trained BERT model.
[0015] Furthermore, in step S2, the specific processing steps of the feature interaction fusion module include:
[0016] The original feature vector is projected onto a common embedding space to obtain the projected feature vector;
[0017] The projected feature vector is input into a nonlinear network to generate an intermediate representation;
[0018] Based on the intermediate representation, independent attention weights are generated for the projected feature vector;
[0019] The attention weights are applied to the corresponding projected feature vectors, and the results are merged to form the deep fusion feature vector.
[0020] Further, in step S4, the weight parameters of the cost-sensitive loss function are calculated based on the inverse frequency of the number of samples in each category, and the weight of each category is determined by the following formula: total number of samples / (total number of categories * number of samples in that category).
[0021] Based on the same inventive concept, this application also provides a Monteggia fracture intelligent classification device based on image-text multimodal fusion used in the above-mentioned method, comprising:
[0022] The image feature extraction unit is used to generate the original image feature vector using a pre-trained ViT model;
[0023] The text feature extraction unit is used to generate raw text feature vectors using a pre-trained BERT model.
[0024] The feature interaction fusion unit is used to generate a deep fusion feature vector based on the original image feature vector and the original text feature vector.
[0025] The hierarchical feature aggregation unit is used to concatenate the deep fusion feature vector with the original image feature vector and the original text feature vector to form a hierarchical aggregated feature vector;
[0026] A classification unit is used to receive the hierarchical aggregated feature vector and output the classification result;
[0027] The training optimization module is configured to update the parameters of the classification unit using a cost-sensitive loss function during model training. The weight parameters of the loss function are inversely proportional to the frequency of each class of samples in the training dataset.
[0028] Furthermore, the feature interaction fusion unit further includes:
[0029] The projection module is used to map the original feature vectors to a common embedding space;
[0030] Nonlinear interactive networks are used to generate an intermediate representation from projected feature vectors;
[0031] Multiple weight generation networks are used to generate attention weights from the intermediate representation for weighted projected feature vectors.
[0032] The beneficial effects of this invention are:
[0033] The intelligent Monteggia fracture classification method based on image-text multimodal fusion provided by this invention deeply integrates radiographic X-ray images with clinical text descriptions. In computer-aided diagnosis, it accurately simulates the comprehensive diagnostic logic and thinking patterns of top orthopedic surgeons. Utilizing a feature interaction fusion module, it effectively uncovers hidden correlation features that are difficult to detect under a single modality, significantly improving the identification accuracy of difficult cases (especially those with obvious ulnar fractures but concealed radial head dislocation), fundamentally reducing the risk of missed diagnoses. Simultaneously, by integrating a cost-sensitive learning strategy, it applies higher misclassification penalty weights to rare Monteggia fracture classifications during model training, ensuring that the model possesses a high level of diagnostic and identification capabilities for all types of classifications (including rare types), thereby achieving comprehensive and accurate coverage of the entire Monteggia fracture disease spectrum. Attached Figure Description
[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0035] Figure 1 This is a schematic diagram of the model architecture in an embodiment of the present invention;
[0036] Figure 2 This is a multi-class ROC curve diagram of four types of Monteggia fracture in an embodiment of the present invention;
[0037] Figure 3 This is a model confusion matrix diagram in an embodiment of the present invention. Detailed Implementation
[0038] To facilitate understanding of the present invention, it will be described more fully below through embodiments, and preferred embodiments are given below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Any other implementation schemes obtained by modifying or equivalently substituting the technical solutions of the present invention without inventive step are all within the protection scope of the present invention.
[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0040] The numerical values disclosed in the embodiments of this invention are approximate values, not definitive values. Where error or experimental conditions permit, all values within the error range may be included, and the specific numerical values disclosed in the embodiments of this invention are not limited to those specified.
[0041] Unless otherwise specified, all instruments and equipment used in this invention can be purchased from the market or prepared by existing methods.
[0042] See Figure 1 As shown, this embodiment provides an intelligent classification method for Monteggia fractures based on image-text multimodal fusion, including the following steps:
[0043] S1. Extract the image data and text data from the input sample respectively, and generate an original image feature vector and an original text feature vector;
[0044] The specific steps are as follows:
[0045] The system employs two parallel pre-trained encoders. First, a VisionTransformer model is used. To process the input image batch I, extract its global [CLS] token output as image features. This feature It summarizes the global visual content of the image. Simultaneously, a BERT model is used. To process the segmented text T (containing input_ids and attention_mask), extract its [CLS] token as the output of the text feature. This feature It summarizes the core semantics of the text.
[0046] S2. Based on the original image feature vector and the original text feature vector, a deep fusion feature vector is generated through a feature interaction fusion module;
[0047] The specific steps are as follows:
[0048] Due to the feature dimensions of the outputs of the two pre-trained encoders, ViT and BERT and They may differ, but by linear transformation they can be projected into a shared embedding space of a uniform dimension. Specifically, this is due to two independent linear layers, namely... and , respectively and Projection as and This makes both of them have dimension.
[0049]
[0050] S3. The deep fusion feature vector, the original image feature vector, and the original text feature vector are concatenated to form a hierarchical aggregated feature vector;
[0051] The specific steps are as follows:
[0052] First, the two projected features are concatenated. The concatenated features are then fed into a multilayer perceptron (MLP) to learn a shared signal that summarizes cross-modal interaction information. The signal S is then used to generate attention weights (gating) for the image and text respectively through two independent linear layers. and Finally, these weights are applied to their respective features using element-wise multiplication, and the weighted features are summed to obtain the final deep fusion features. :
[0053]
[0054] The deep fusion features obtained above Compared with the original image features obtained in step 1 and original text features The features are concatenated along the feature dimension. This results in the final aggregated feature. Its total dimension is It provides the most comprehensive information view for downstream classifiers.
[0055] S4. Input the hierarchical aggregated feature vector into a classifier, and during model training, use a cost-sensitive loss function to optimize the classifier, wherein the weight parameters of the loss function are predetermined based on the distribution of samples of each class in the training dataset.
[0056] The specific steps are as follows:
[0057] To improve training stability, the hierarchical aggregated feature vector is input into a classifier. Layer normalization is performed. The normalized features are then fed into a classification MLP containing a ReLU activation function and a Dropout regularization layer, and finally output through a linear layer. The raw scores (Logits) O for each category are used for subsequent loss calculation.
[0058] The loss function used is the weighted cross-entropy loss function. .in It is a weight vector pre-calculated based on the number of samples in each category in the training set. It makes the category with fewer samples have a higher weight when calculating the total loss, thereby guiding the model to pay more attention to the correct classification of these few categories.
[0059] The implementation parameters for this embodiment are configured as follows:
[0060] 1. Hardware Configuration
[0061] Executed by NVIDIA GeForce RTX 4090 GPU.
[0062] 2. Model Optimization Strategy
[0063] The training cycle is set to 150 epochs per round.
[0064] The optimal model weights are retained in each round (based on the highest weighted F1 score).
[0065] Model optimization was performed using the AdamW optimizer.
[0066] 3. Dataset Description
[0067] In this embodiment, a multimodal dataset containing 477 cases of Monteggia fractures was constructed. Each sample consists of an X-ray image and its corresponding radiological description text. This dataset comprehensively covers all subtypes of Monteggia fractures: type I (122 cases), type II (45 cases), type III (251 cases), and type IV (59 cases).
[0068] To ensure the clinical value and high quality of the data, we implemented a rigorous governance process. Fracture classification for all samples was completed by a team of senior orthopedic attending physicians to guarantee the authority and accuracy of the annotations. Classification strictly adhered to the clinical criteria for Monteggia fractures. During data preprocessing, we systematically excluded low-quality images with technical defects such as insufficient contrast and motion artifacts, as well as samples containing confounding factors such as internal fixation implants or a history of previous surgery. Finally, all qualified images were standardized and cropped, retaining only the core fracture area to eliminate interference from irrelevant anatomical background.
[0069] To avoid model bias due to imbalanced sample classes, we employed a stratified sampling strategy, strictly dividing the dataset into a training set (332 cases) and a test set (145 cases) in a 7:3 ratio. This strategy ensured that the distribution ratio of the four fracture subtypes remained consistent between the training and test sets. Furthermore, to improve the model's generalization ability and robustness, we applied data augmentation techniques, including brightness / saturation adjustment and random rotation, to the image data during the training phase.
[0070] 4. Experimental Verification
[0071] To objectively and comprehensively evaluate the effectiveness of the classification method provided in this embodiment, this application conducted a series of rigorous quantitative comparative experiments and ablation analyses. All experiments were conducted on a unified internal dataset containing both image and text heterogeneous data modalities, and adopted an industry-recognized and comprehensive evaluation index system, including accuracy, precision, recall, and F1 score, to ensure the scientific rigor and impartiality of the evaluation results.
[0072] To demonstrate the significant advancements of this invention compared to existing technologies, we selected several representative baseline models covering different technical approaches for performance benchmarking. These models include: classic convolutional neural network architectures, such as the VGG and ResNet series; advanced visual Transformer models based on self-attention mechanisms, such as ViT and DeiT; and pre-trained models with broad influence in the multimodal domain, such as CLIP and ALIGN.
[0073] The experimental results are shown in Tables 1 and 2.
[0074] Table 1
[0075] Model accuracy Accuracy Recall rate F1 score VGG16 0.7448 0.7391 0.7448 0.7023 VGG19 0.6966 0.6595 0.6966 0.6466 ResNet18 0.7517 0.7444 0.7517 0.7220 ResNet34 0.7655 0.7416 0.7655 0.7423 ResNet50 0.7655 0.7041 0.7655 0.7052 ResNet101 0.8000 0.8265 0.8000 0.7660 DenseNet121 0.7517 0.7807 0.7517 0.7151 DenseNet169 0.7448 0.6910 0.7448 0.6998 MobileNet_V2 0.7241 0.6411 0.7241 0.6595 MobileNet_V3_Small 0.7379 0.7278 0.7379 0.7128 EfficientNet_B0 0.6276 0.4894 0.6276 0.5354 EfficientNet_B1 0.7517 0.6729 0.7517 0.7067 ViT_Ti 0.7241 0.7475 0.7241 0.6920 ViT_B 0.8000 0.6829 0.6351 0.6351 Deit_Ti 0.7862 0.7833 0.7862 0.7773 Deit-S 0.7655 0.7639 0.7655 0.7537 Deit-B 0.7724 0.7489 0.7724 0.7501 EfficientFormer_L1 0.5862 0.4928 0.5862 0.5318 EfficientFormer_L3 0.6207 0.4822 0.6207 0.5420 Ours 0.8414 0.832 0.8414 0.8345
[0076] Table 2
[0077] Model accuracy Accuracy Recall rate F1 score CLIP 0.8069 0.8012 0.8069 0.7866 ALIGN 0.5586 0.4955 0.5586 0.511 LXMERT 0.7655 0.7623 0.7655 0.7505 UNITER 0.7517 0.7359 0.7517 0.7347 LGAD 0.8138 0.8132 0.8138 0.7928 Co-Attention 0.7862 0.7609 0.7862 0.7658 Ours 0.8414 0.832 0.8414 0.8345
[0078] Referring to the test results in Tables 1 and 2 above, the data clearly demonstrates that the model proposed in this invention exhibits comprehensive performance advantages across all five core evaluation metrics, significantly outperforming all baseline models compared. Specifically, the model of this invention achieves 84.14% accuracy, 83.2% precision, 84.14% recall, and 83.45% F1 score.
[0079] In terms of the F1 score, a comprehensive measure of classification performance, the present invention achieves a relative improvement of 2.76% (0.8345) compared to the second-best performing LGAD model (0.8138), representing a substantial advancement on a highly optimized baseline. This result strongly demonstrates that the model of the present invention not only classifies accurately but also possesses stronger robustness and more reliable generalization ability, effectively distinguishing different categories. This performance advantage stems from the unique feature extraction and deep fusion mechanism of the present invention, which can capture and utilize cross-modal semantic association information more effectively than existing technologies.
[0080] To further verify the indispensability and design rationality of the core modules of the classification method of this invention, we conducted an in-depth analysis of the model through ablation experiments. The test results are detailed in Table 3. This study aims to isolate and quantify the contribution of each part of the model to the final performance.
[0081] Table 3
[0082] Model accuracy Accuracy Recall rate F1 score Image Only 0.7724 0.7825 0.7724 0.7715 Text Only 0.4966 0.5599 0.4966 0.5173 Simple Concat Fusion 0.7931 0.8122 0.7931 0.8006 Full Model 0.8138 0.8053 0.8138 0.7976 Full Model plus loss 0.8414 0.832 0.8414 0.8345
[0083] By comparing the performance of unimodal models ("Image Only") and multimodal fusion models ("Text Only"), we validated the necessity of information fusion. Data shows that the performance of unimodal models, such as the "Image Only" model with an F1 score of 0.7715, is significantly lower than any fusion scheme. This confirms that information from a single source is insufficient to complete complex classification tasks, while combining the visual content of images with the semantic description of text generates information gain, which is a key prerequisite for improving performance.
[0084] Table 3 shows a comparison between the "Full Model" of this invention and the model using the "Simple ConcatFusion" strategy. The results show that although simple concatfusion also improves performance (F1 score of 0.8006), the "Full Model" using the feature interaction fusion module proposed in this invention outperforms the former in multiple metrics, including accuracy and recall. This decisively proves that the fusion module of this invention is not simply feature stacking, but rather achieves a deeper and more adaptive interaction and alignment between image and text features through its internal nonlinear transformation and attention weighting mechanism, thereby generating a higher-quality, more discriminative multimodal fusion representation. Comparing the "Full Model" with the final solution (Full Model + loss) applying a specific loss optimization strategy, the final solution achieves peak performance on all metrics. This indicates that this invention is not only innovative in its model architecture, but its accompanying training and optimization strategies are also an indispensable part of the overall technical solution. The two work synergistically to maximize the technical effect.
[0085] See also: Figure 2 The multi-class ROC curves in this application show that the model exhibits superior performance across most categories. Specifically, the model demonstrates exceptional discriminative ability for Type 1, Type 3, and Type 4 categories, with an AUC value of 0.96 for each category. This means the model can almost perfectly separate these categories from others. However, the model's ability to identify Type 2 categories is slightly weaker, with an AUC value of 0.80. Nevertheless, this result is still significantly better than random guessing, indicating that the model possesses a certain level of discriminative ability.
[0086] See also: Figure 3 The figures show the performance of our model on the overall classification task of four types of Monteggia fractures, demonstrating high overall accuracy. It performs particularly well in identifying type III fractures, with very high recall and precision, accurately identifying the vast majority of type III fractures from the samples. The classification of types I and IV fractures also shows good results, demonstrating high recognition ability. However, it is slightly weaker in identifying type II fractures, with a recall rate of only about 35.7%. Overall, our model has strong diagnostic capabilities for common fracture types.
[0087] In summary, the classification method and apparatus provided by this invention are specifically designed for the automatic identification of Monteggia fracture types. This method innovatively integrates X-ray images and fracture text information, effectively improving classification capabilities in complex medical images. Experimental verification shows that this invention exhibits excellent and robust performance in multiple key indicators, including accuracy, recall, and performance under different detection difficulties, fully demonstrating its reliability and effectiveness as a highly efficient automated diagnostic tool for Monteggia fracture classification, providing a reliable basis for precision treatment such as Monteggia fracture classification.
[0088] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. An intelligent Meng's fracture typing method based on image-text multi-modal fusion, characterized in that: The method comprises the following steps: S1. Extracting image data and text data in the input sample respectively to generate an original image feature vector and an original text feature vector; S2. Generating a deep fusion feature vector through a feature interaction fusion module according to the original image feature vector and the original text feature vector; S3. Concatenating the deep fusion feature vector, the original image feature vector and the original text feature vector to form a hierarchical aggregation feature vector; S4. Inputting the hierarchical aggregation feature vector into a classifier, and using a cost-sensitive loss function to optimize the classifier during model training, wherein the weight parameter of the loss function is determined according to the distribution of samples of each category in the training data set.
2. The intelligent Meng's fracture typing method based on image-text multi-modal fusion according to claim 1, characterized in that, In step S1, the original image feature vector is extracted by a pre-trained ViT model, and the original text feature vector is extracted by a pre-trained BERT model.
3. The intelligent Meng's fracture typing method based on image-text multi-modal fusion according to claim 1, characterized in that, In step S2, the specific processing steps of the feature interaction fusion module include: Projecting the original feature vector to a common embedding space to obtain a projected feature vector; Inputting the projected feature vector into a nonlinear network to generate an intermediate representation; Generating independent attention weights for the projected feature vector based on the intermediate representation; Applying the attention weights to the corresponding projected feature vector and merging the results to form the deep fusion feature vector.
4. The intelligent Meng's fracture typing method based on image-text multi-modal fusion of claim 1, characterized in that, In step S4, the weight parameter of the cost-sensitive loss function is calculated according to the inverse frequency of the number of samples of each category, and the weight of each category is determined by the following formula: total sample number / (total number of categories * sample number of the category).
5. The intelligent Meng's fracture typing device based on the fusion of multi-modal of graphs and texts for use in the method of any one of claims 1-4, characterized in that, It comprises: An image feature extraction unit for generating an original image feature vector using a pre-trained ViT model; A text feature extraction unit for generating an original text feature vector using a pre-trained BERT model; A feature interaction fusion unit for generating a deep fusion feature vector according to the original image feature vector and the original text feature vector; A hierarchical feature aggregation unit for concatenating the deep fusion feature vector, the original image feature vector and the original text feature vector to form a hierarchical aggregation feature vector; A classification unit for receiving the hierarchical aggregation feature vector and outputting a classification result; A training optimization module configured to update the parameters of the classification unit using a cost-sensitive loss function during model training, and the weight parameter of the loss function is inversely proportional to the frequency of samples of each category in the training data set.
6. The intelligent Mendes fracture typing device based on image-text multi-modal fusion of claim 5, wherein, The feature interaction fusion unit further comprises: A projection module for mapping the original feature vector to a common embedding space; A nonlinear interaction network for generating an intermediate representation from the projected feature vector; A plurality of weight generation networks for generating attention weights for weighting the projected feature vector from the intermediate representation.