Early diagnosis system and method of knee arthritis based on multimodal deep learning

Abstract

The present invention relates to the technical field of medical diagnostic systems, and specifically to an early diagnosis system and method for knee arthritis based on multimodal deep learning. The data collection module is used to collect and preprocess multimodal data including knee joint X-ray image data, MRI image data, and clinical evaluation data. The model construction module constructs a deep learning model by combining a convolutional neural network and a Transformer architecture, and uses a feature automatic learning module and a potential relationship mining module to achieve mapping from multimodal data to clinical evaluation phenotype labels. At the same time, the Grad-CAM interpretability method is integrated to understand the basis of model prediction, and diagnostic data is continuously collected and fed back to the data collection module for model optimization. The convolutional network is good at capturing local image features, while the Transformer architecture is good at modeling long-range dependencies. The combination of the two greatly improves the feature extraction and relationship modeling capabilities of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of medical diagnostic systems, and in particular to an early diagnosis system and method for knee arthritis based on multimodal deep learning. Background Art

[0002] In recent years, with the aging of the population, knee arthritis has become a common disease that seriously affects the quality of life of middle-aged and elderly people. Currently, clinical diagnosis mainly relies on doctors based on the results of imaging examinations such as X-rays and MRIs. However, this traditional diagnostic method has many limitations: First, because the early symptoms of knee arthritis are relatively subtle, single-modality imaging examinations often have difficulty accurately identifying the characteristics of early lesions. Second, different doctors may reach different diagnostic conclusions when interpreting the same set of images, which is highly subjective. Third, traditional diagnostic methods cannot effectively predict the progression of the disease, making timely intervention difficult.

[0003] While recent research has attempted to apply deep learning technology to the diagnosis of knee arthritis, existing approaches still face numerous technical bottlenecks. For one thing, most studies focus solely on analyzing single-modality imaging data, overlooking the potential for deeper connections between multimodal data. Furthermore, current deep learning models often suffer from a "black box" problem, meaning they are unable to explain the specific diagnostic rationale to physicians. This severely restricts their widespread application in clinical practice. Furthermore, existing systems generally lack mechanisms for continuous optimization, making them difficult to adapt to evolving clinical needs.

[0004] More importantly, existing technical solutions often focus on improving model accuracy while ignoring the actual needs of doctors during the diagnostic process. For example, many systems only output simple binary classification results, without providing detailed diagnostic explanations or risk warnings. Furthermore, the systems' poor interactivity prevents doctors from intuitively understanding the model's decision-making process, which, to a certain extent, limits the practical value of AI-assisted diagnosis systems. Summary of the Invention

[0005] This invention aims to overcome these technical bottlenecks by proposing a multimodal deep learning-based early diagnosis system and method for knee arthritis. By innovatively integrating three functional modules: data collection, model building, and system deployment, this system enables intelligent diagnosis throughout the entire process, from data processing to clinical application.

[0006] The present invention proposes an early diagnosis system for knee arthritis based on multimodal deep learning, including a data collection module, a model construction module and a system deployment module, wherein the data collection module is used to collect and preprocess multimodal data including knee joint X-ray image data, MRI image data and clinical evaluation data; the model construction module constructs a deep learning model by combining convolutional neural networks and Transformer architecture, and adopts a feature automatic learning module and a potential relationship mining module to realize the mapping from multimodal data to clinical evaluation phenotype labels, while integrating the Grad-CAM interpretability method to understand the basis of model prediction; the system deployment module is responsible for deploying the diagnostic system and providing online services, and continuously collects diagnostic data to feed back to the data collection module for model optimization.

[0007] Preferably, the data collection module first obtains a multimodal data set including knee joint X-ray image data, MRI image data and corresponding clinical evaluation data from the hospital database, and then uses the Pydicom toolkit to decode the knee joint X-ray image data into DICOM files and convert the format to generate a standard image with a resolution of 128×128. The MRIcro toolkit is then used to convert the MRI image data into NIfTI format and set a region of interest mask to retain the pixel values ​​of key areas. Finally, the clinical evaluation data is obtained from the CCKS2017 database through crawler technology, and all preprocessed data are integrated into a standardized data set.

[0008] Preferably, the feature automatic learning module in the model construction module adopts a two-layer convolutional neural network structure with n×m convolution kernels to extract n×m dimensional feature vectors from multimodal image data, and the potential relationship mining module adopts a Transformer architecture with h attention heads to realize multi-dimensional analysis of the relationship between features by setting the query matrix, key matrix and value matrix, and finally merges the analysis results of multi-head attention through the weighted average method.

[0009] Preferably, the model building module adopts two different model architectures according to the number of convolutional layers k. When k=1, a hybrid model architecture is used to use the output of the convolutional neural network as the input of the Transformer. When k>1, the Transformer architecture is used alone and the output results of the feature automatic learning module are input into the multi-head attention mechanism for further analysis.

[0010] Preferably, the model building module achieves the interpretability of the model through the following steps: first, the prediction results are preprocessed using the soft labeling method, then the convolutional layer feature map is back-propagated to obtain the weight gradient, then the weight gradient is multiplied by the soft label to generate an attention map, and finally the attention map is multiplied by the original feature map to obtain the final Grad-CAM visualization result.

[0011] Preferably, the model building module optimizes the model parameters using a loss function including a cross entropy loss term and a regularization term, controls the parameter update speed by setting the learning rate and update step size, and evaluates the model performance after each round of training to decide whether to continue optimization.

[0012] Preferably, the system deployment module includes a knee arthritis diagnosis module and a knee arthritis prediction module, wherein the knee arthritis diagnosis module first receives multimodal image input, then calculates the diagnosis probability through a deep learning model, and finally outputs the diagnosis result based on a preset threshold; the knee arthritis prediction module analyzes the disease development trend based on historical data and outputs the risk level.

[0013] A method for an early diagnosis system for knee arthritis based on multimodal deep learning includes the following steps: first, acquiring and preprocessing multimodal data through the data collection module, then training a deep learning model using the model construction module, and finally deploying the system deployment module to provide online diagnosis services.

[0014] Preferably, the method specifically includes the following steps: when a user uploads multimodal imaging data for diagnosis, the system first outputs a normal or abnormal diagnosis result through the knee arthritis diagnosis module, and then outputs a low-risk or high-risk prediction result through the knee arthritis prediction module. When the diagnosis result is abnormal or the prediction result is high-risk, the system automatically generates and displays the Grad-CAM visualization result.

[0015] Preferably, when the diagnosis result is normal or the prediction result is low risk, the system displays the diagnosis basis through the following steps: first, calculate and filter the top 10 image features with the highest importance weights, and then display these features in the form of interactive icons, where the user can view the feature description and mark the corresponding area in the original image with a rectangular box when hovering the mouse over the icon. When the diagnosis result is abnormal, the system simultaneously displays the classification result, confidence score and explanatory image.

[0016] The beneficial technical effects brought about by the technical solution of the present invention are:

[0017] At the architectural design level, the present invention adopts a modular design approach, forming an organic closed loop between the functional modules. The data collection module provides high-quality multimodal data for model training through a standardized preprocessing process. The model construction module is based on an innovative hybrid deep learning architecture, which realizes the deep fusion and intelligent analysis of multimodal data. The system deployment module not only provides convenient online diagnostic services, but also continuously collects clinical feedback to optimize model performance. This closed-loop design ensures that the system can continuously improve itself and adapt to various needs in clinical practice.

[0018] In terms of algorithmic innovation, this invention breaks new ground by combining convolutional neural networks with the Transformer architecture, leveraging the strengths of both models. Convolutional networks excel at capturing local image features, while the Transformer architecture excels at modeling long-range dependencies. This combination significantly enhances the model's feature extraction and relationship modeling capabilities. Furthermore, by integrating an improved Grad-CAM visualization method, the system generates intuitive interpretable results, helping doctors understand the model's diagnostic rationale.

[0019] In terms of practicality, this invention features targeted optimizations in multiple key areas: First, by setting appropriate diagnostic thresholds and risk levels, the system can provide differentiated diagnostic recommendations for patients in different situations. Second, an innovative interactive result display allows doctors to more intuitively understand and verify diagnostic results. Third, the system's continuous optimization mechanism ensures that diagnostic performance continues to improve as data accumulates.

[0020] The technical solution of this invention not only solves several key issues existing in existing technologies but also demonstrates significant benefits in practical applications: significantly improved diagnostic accuracy, enhanced interpretability, and significantly improved system usability. In particular, this invention demonstrates unique technical advantages in identifying early lesions and predicting disease progression, providing strong technical support for advancing the diagnosis and treatment of knee arthritis.

[0021] It's important to emphasize the close synergy between the various technical features of this invention. For example, the integration of multimodal data not only improves diagnostic accuracy but also provides a richer source of information for interpretable analysis. The system's continuous optimization mechanism further enhances the model's generalization capabilities through the continuous accumulation of clinical data. The organic combination of these technical features ultimately forms a high-performance, reliable, and stable intelligent diagnostic system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the system logic block diagram of the present invention.

[0023] Figure 2 This is a diagram of the deep learning model architecture of the present invention. DETAILED DESCRIPTION

[0024] To further illustrate the technical means and effects employed by the present invention to achieve its intended objectives, the following detailed description, along with the accompanying drawings and preferred embodiments, includes a detailed description of the specific implementations, structures, features, and effects thereof. In the following description, references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0025] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0026] See Figure 1-2 The present invention provides an early diagnosis system for knee arthritis based on multimodal deep learning, comprising a data collection module 1, a model building module 2, and a system deployment module 3. The data collection module 1 is primarily responsible for acquiring and preprocessing knee joint X-ray image data, MRI image data, and clinical evaluation data. The model building module 2 achieves intelligent mapping from multimodal data to clinical evaluation phenotype labels by combining convolutional neural networks and the Transformer architecture. The system deployment module 3 is responsible for deploying the diagnostic system and providing online services. Preferably, the system deployment module 3 also has the function of continuously collecting diagnostic data and feeding it back to the data collection module 1, thereby achieving continuous optimization of the diagnostic model.

[0027] In a preferred embodiment of the present invention, the specific workflow of data collection module 1 is as follows: First, a complete multimodal dataset, including knee X-ray images, MRI images, and corresponding clinical assessment data, is obtained from a hospital database. The X-ray images are then processed using the Pydicom toolkit, which decodes DICO M format files and then converts the images to a standard resolution of 128×128 pixels. In a preferred embodiment of the present invention, a resolution of 128×128 is chosen because it preserves sufficient diagnostic information while effectively controlling computational complexity. The MRI data is then processed using the MRIcro toolkit and converted to the standard NIfTI format. Notably, a region of interest mask is set during the conversion process to focus the analysis on key diagnostic areas such as the knee cartilage and meniscus. Finally, standardized clinical assessment data is obtained from authoritative databases such as CCKS2017 using crawler technology.

[0028] In a preferred embodiment of the present invention, the model building module 2 of the present invention includes two core components: a feature automatic learning submodule 21 and a potential relationship mining submodule 22. The feature automatic learning submodule 21 adopts a two-layer convolutional neural network structure with n×m convolution kernels to extract n×m-dimensional feature vectors from multimodal images. Preferably, n and m are 64 and 128, respectively, which can ensure the adequacy of feature extraction without causing excessive computational burden. The potential relationship mining submodule 22 adopts a Transformer architecture with h attention heads (preferably, h=8), and realizes multi-dimensional analysis of the relationship between features by setting the query matrix Q, key matrix K and value matrix V.

[0029] In a preferred embodiment of the present invention, model construction module 2 employs different model architecture strategies depending on the number of convolutional layers, k. Specifically, when k = 1, a hybrid model architecture is employed, directly feeding the output features of the convolutional neural network into the Transformer for further processing. When k > 1, feature extraction and relationship modeling are performed entirely within the Transformer architecture. Extensive experiments have verified that this adaptive architecture selection strategy achieves good results in various scenarios.

[0030] In a preferred embodiment of the present invention, an innovative visualization method based on Grad-CAM is integrated. The specific implementation steps are as follows: First, soft-label preprocessing is performed on the prediction results, preferably multiplying the predicted probability by a coefficient of 0.8 to prevent zero values ​​from affecting subsequent processing. Then, a backpropagation operation is performed on the convolutional layer feature maps to obtain the importance weights of each feature map. The weights are then multiplied by the soft labels to generate an attention map. Finally, the attention map is multiplied with the original feature map to obtain the final Grad-CAM visualization result.

[0031] In a preferred embodiment of the present invention, an improved loss function is used for parameter optimization during model training. The loss function includes a cross-entropy loss term and an L2 regularization term, with the regularization coefficient preferably set to 0.001. The parameter update strategy uses the Adam optimizer for gradient descent, with an initial learning rate of 0.001. The learning rate is then reduced to 0.1 after validation set performance stops improving for five consecutive epochs.

[0032] In a preferred embodiment of the present invention, the system deployment module 3 comprises two main functional units: a diagnostic submodule 31 and a prediction submodule 32. After receiving user-uploaded images, the diagnostic submodule 31 first performs standardized preprocessing and then inputs them into a deep learning model to calculate the diagnostic probability. In a preferred embodiment of the present invention, two diagnostic thresholds, 0.3 and 0.7, are set: when the diagnostic probability is greater than 0.7, it is considered abnormal; when it is less than 0.3, it is considered normal; and when it is between 0.3 and 0.7, further examination is recommended. The prediction submodule 32 analyzes the risk of disease progression based on historical data, with the high-risk warning threshold preferably set at 0.8.

[0033] This invention also discloses a method for an early diagnosis system for knee arthritis based on multimodal deep learning. The specific implementation process of this invention is divided into three main stages. The first is the data preprocessing stage, where a complete multimodal dataset is acquired and organized using the data collection module 1. The second is the model training stage, where a deep learning model is built and optimized using the model construction module 2. Finally, the system deployment stage occurs, where the system deployment module 3 is activated to provide online diagnostic services.

[0034] In a preferred embodiment of the present invention, when a user seeks a diagnosis for knee arthritis, they first upload relevant images through the system's web interface. The system automatically preprocesses the images and, through the diagnosis submodule 31, provides a normal / abnormal diagnosis result. Simultaneously, through the prediction submodule 32, it outputs a low-risk / high-risk risk prediction. Specifically, when the system determines an abnormal or high-risk condition, it automatically invokes the Grad-CAM module to generate explanatory visualizations in the form of heatmaps.

[0035] In a preferred embodiment of the present invention, the present invention adopts a humanized interactive design in the result display link. For normal or low-risk cases, the system will calculate and screen the top 10 image features with the highest importance weights, and present them in the form of interactive icons. When the user hovers the mouse over a feature icon, the color of the icon will deepen (preferably, from light blue #E6F3FF to dark blue #1890FF), and the corresponding area will be marked with a red rectangular frame (line width preferably 2 pixels) on the original image, accompanied by a detailed text description. For abnormal cases, in addition to giving clear classification results, a confidence score between 0 and 1 and an explanatory image based on Grad-CAM will also be displayed.

[0036] In this embodiment of the present invention, when processing knee joint X-ray images, data collection module 1 not only adjusts the resolution to a uniform 128×128 pixel resolution, but also performs the following preprocessing steps: First, the image undergoes histogram equalization, preferably using the CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm, with a contrast limit threshold set to 3.0 and a grid size of 8×8 pixels. Next, image denoising is performed, preferably using a non-local means (Non-local Means) filter algorithm, with a search window size of 21×21 pixels, a similarity window size of 7×7 pixels, and a filter strength h of 10. This effectively improves image quality and facilitates subsequent feature extraction.

[0037] For MRI data preprocessing, the present invention uses the MRIcro toolkit to set the ROI mask, specifically following the steps below: First, based on anatomical knowledge, the knee joint region is divided into multiple regions of interest (ROIs), such as the cartilage region, the meniscus region, and the ligament region. A binary mask is then set for each region, with the pixel value of the ROI set to 1 and all other regions set to 0. Preferably, an atlas-based automatic segmentation method is used for regional segmentation, where the atlas template is derived from the publicly available OAI (Osteoarthritis Initiative) dataset.

[0038] The two-layer convolutional neural network in the automatic feature learning submodule 21 has the following parameters: the first convolutional layer uses 64 3×3 convolution kernels with a stride of 1 and a padding of 1, followed by a ReLU activation function and a 2×2 max pooling layer with a stride of 2. The second convolutional layer uses 128 3×3 convolution kernels, with the same other parameters as the first layer. A batch normalization layer is inserted between the two convolutional layers to help accelerate training and improve model generalization.

[0039] In the latent relationship mining submodule 22, the Transformer architecture is implemented as follows: Input features are first mapped to a 512-dimensional feature space via linear projection, followed by multi-head self-attention computation using eight attention heads. Each attention head has a feature dimension of 64 (512 / 8), and the attention dropout rate is set to 0.1. The feedforward neural network employs a two-layer structure with a hidden layer dimension of 2048 and the ReLU activation function.

[0040] The training process of the present invention employs an improved learning strategy: first, the convolutional neural network is initialized using pretrained ImageNet weights, followed by a two-stage training approach. In the first stage, the convolutional layer parameters are fixed, and only the Transformer layer is trained for 15 epochs. In the second stage, all layers are unfrozen and end-to-end training is performed for 30 epochs. Preferably, the AUC metric is calculated on the validation set after each epoch. If the validation set AUC does not improve after five consecutive epochs, the learning rate is reduced to the original 0.1.

[0041] This paper improves upon the traditional Grad-CAM approach in its interpretability implementation. First, when calculating the class activation map, it not only considers the convolutional feature maps from the last layer but also incorporates the feature maps from the penultimate and third layers. Specifically, these three layers are upsampled to the same spatial resolution and then fused using a weighted average, with weights of 0.5, 0.3, and 0.2, respectively. This approach yields saliency information at multiple scales, improving the accuracy of visualization results.

[0042] During system deployment, the present invention implements a dynamic model update mechanism: the system regularly collects new diagnostic data (preferably every two weeks) and calculates the model's performance on this new data. If the diagnostic accuracy of a particular case type decreases significantly (for example, the average accuracy of 10 consecutive cases of the same type falls below 0.8), the system automatically triggers a model fine-tuning process. During fine-tuning, a small learning rate (preferably 0.0001) is used for incremental learning to avoid undermining the model's good performance on existing data.

[0043] In practical applications, the present invention has demonstrated that the above-mentioned technical solution provides an excellent balance between diagnostic accuracy and computational efficiency. For example, during a six-month clinical trial at a tertiary hospital, the system achieved a 92.3% diagnostic accuracy rate for over 2,000 cases of knee arthritis, with an average single diagnosis time of only 0.8 seconds. Over 95% of the diagnostic results were recognized by the attending physicians.

[0044] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. An early diagnosis system for knee arthritis based on multimodal deep learning, characterized by: The system comprises a data collection module, a model building module and a system deployment module, wherein the data collection module is used to collect and preprocess multimodal data including knee joint X-ray imaging data, MRI imaging data and clinical assessment data; the model building module constructs a deep learning model by combining a convolutional neural network and a Transformer architecture, and uses a feature automatic learning module and a potential relationship mining module to achieve mapping from multimodal data to clinical assessment phenotype labels, while integrating the Grad-CAM interpretability method for understanding the basis of model prediction; the system deployment module is responsible for deploying the diagnostic system and providing online services, and continuously collects diagnostic data to feed back to the data collection module for model optimization; The feature automatic learning module in the model construction module uses a two-layer convolutional neural network structure with n×m convolution kernels to extract n×m dimensional feature vectors from multimodal image data. The potential relationship mining module uses a Transformer architecture with h attention heads to achieve multi-dimensional analysis of the relationship between features by setting a query matrix, a key matrix, and a value matrix. Finally, the analysis results of the multi-head attention are combined through a weighted average method. The model building module adopts two different model architectures depending on the number of convolutional layers k. When k=1, a hybrid model architecture is used to take the output of the convolutional neural network as the input of the Transformer. When k>1, the Transformer architecture is used alone and the output of the feature automatic learning module is input into the multi-head attention mechanism for further analysis. The model building module achieves model interpretability through the following steps: first, the prediction results are preprocessed using the soft labeling method, then the convolutional layer feature map is back-propagated to obtain the weight gradient, then the weight gradient is multiplied by the soft label to generate the attention map, and finally the attention map is multiplied by the original feature map to obtain the final Grad-CAM visualization result; The system deployment module includes a knee arthritis diagnosis module and a knee arthritis prediction module. The knee arthritis diagnosis module first receives multimodal image input, then calculates the diagnosis probability through a deep learning model, and finally outputs the diagnosis result based on a preset threshold. The knee arthritis prediction module analyzes the disease development trend based on historical data and outputs the risk level.

2. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 1, characterized in that: The data collection module first obtains a multimodal dataset containing knee joint X-ray image data, MRI image data and corresponding clinical evaluation data from the hospital database, then uses the Pydicom toolkit to decode and convert the DICOM file of the knee joint X-ray image data to generate a standard image with a resolution of 128×128. The MRIcro toolkit is then used to convert the MRI image data into NIfTI format and set a region of interest mask to retain the pixel values ​​of key areas. Finally, the clinical evaluation data is obtained from the CCKS2017 database through crawler technology, and all preprocessed data are integrated into a standardized dataset.

3. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 1, characterized in that: The model building module uses a loss function containing a cross-entropy loss term and a regularization term to optimize the model parameters, controls the parameter update speed by setting the learning rate and update step size, and evaluates the model performance after each round of training to decide whether to continue optimization.

4. The method for the early diagnosis system of knee arthritis based on multimodal deep learning according to any one of claims 1 to 3, characterized in that: The method comprises the following steps: first, acquiring and preprocessing multimodal data through the data collection module, then training a deep learning model using the model construction module, and finally deploying the system deployment module to provide online diagnostic services.

5. The method according to claim 4, characterized in that The specific steps include: When a user uploads multimodal imaging data for diagnosis, the system first outputs a normal or abnormal diagnosis result through the knee arthritis diagnosis module, and then outputs a low-risk or high-risk prediction result through the knee arthritis prediction module. When the diagnosis result is abnormal or the prediction result is high-risk, the system automatically generates and displays the Grad-CAM visualization result.

6. The method according to claim 5, characterized in that When the diagnosis is normal or the prediction is low-risk, the system displays the diagnostic basis through the following steps: first, it calculates and filters the top 10 image features with the highest importance weights, and then displays these features in the form of interactive icons. When the user hovers the mouse over the icon, they can view the feature description and mark the corresponding area in the original image with a rectangular box. When the diagnosis is abnormal, the system simultaneously displays the classification result, confidence score, and explanatory image.

Images