Early diagnosis system and method for gonitis based on multi-modal deep learning

Abstract

The invention relates to the technical field of medical diagnosis systems, in particular to an early gonitis diagnosis system and method based on multi-modal deep learning, a data collection module is used for collecting and preprocessing multi-modal data including knee joint X-ray image data, MRI (Magnetic Resonance Imaging) image data and clinical evaluation data; the model construction module constructs a deep learning model by combining a convolutional neural network and a Transform architecture, realizes mapping from multi-modal data to a clinical assessment phenotype label by adopting a feature automatic learning module and a potential relationship mining module, and integrates a Grad-CAM interpretability method for understanding a model prediction basis; diagnostic data are continuously collected to be fed back to the data collection module for model optimization, the convolutional network is good at capturing local image features, the Transform architecture is good at modeling a long-range dependency relationship, and the feature extraction and relationship modeling capabilities of the model are greatly improved through combination of the convolutional network and the Transform architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of medical diagnosis systems, and in particular to an early diagnosis system and method for knee arthritis based on multimodal deep learning. Background Art

[0002] In recent years, with the aging of the population, knee arthritis has become a common disease that seriously affects the quality of life of middle-aged and elderly people. At present, the clinical diagnosis mainly relies on doctors based on the results of imaging examinations such as X-rays and MRI. However, this traditional diagnostic method has many limitations: First, due to the relatively hidden early symptoms of knee arthritis, single-modality imaging examinations often have difficulty in accurately identifying early lesion characteristics. Secondly, different doctors may come to different diagnostic conclusions when interpreting the same set of images, which is highly subjective. Thirdly, traditional diagnostic methods cannot effectively predict the progression of the disease and it is difficult to intervene in time.

[0003] Although recent studies have attempted to apply deep learning technology to the diagnosis of knee arthritis, the existing solutions still have many technical bottlenecks: on the one hand, most studies only focus on the analysis of single-modality imaging data, ignoring the deep correlation that may exist between multimodal data. On the other hand, current deep learning models generally have the "black box" problem, that is, they cannot explain the specific diagnostic basis to doctors, which seriously restricts their promotion and application in clinical practice. In addition, the existing systems generally lack a continuous optimization mechanism and are difficult to adapt to changing clinical needs.

[0004] More importantly, existing technical solutions often focus on improving the accuracy of the model, while ignoring the actual needs of doctors in the diagnosis process. For example, many systems only output simple binary classification results without providing detailed diagnostic explanations and risk warnings. At the same time, the system's interactivity is poor, and doctors cannot intuitively understand the decision-making process of the model, which to some extent limits the practical value of AI-assisted diagnosis systems. Summary of the invention

[0005] The present invention aims to break through the above technical bottlenecks and proposes an early diagnosis system and method for knee arthritis based on multimodal deep learning. The system innovatively integrates three functional modules: data collection, model building, and system deployment, and realizes full-process intelligent diagnosis from data processing to clinical application.

[0006] The present invention proposes an early diagnosis system for knee arthritis based on multimodal deep learning, including a data collection module, a model construction module and a system deployment module, wherein the data collection module is used to collect and preprocess multimodal data including knee joint X-ray image data, MRI image data and clinical evaluation data; the model construction module constructs a deep learning model by combining a convolutional neural network and a Transformer architecture, and uses a feature automatic learning module and a potential relationship mining module to achieve mapping from multimodal data to clinical evaluation phenotype labels, while integrating a Grad-CAM interpretability method for understanding the basis of model prediction; the system deployment module is responsible for deploying the diagnosis system and providing online services, and continuously collects diagnosis data to feed back to the data collection module for model optimization.

[0007] Preferably, the data collection module first obtains a multimodal data set including knee joint X-ray image data, MRI image data and corresponding clinical evaluation data from the hospital database, and then uses the Pydicom toolkit to perform DICOM file decoding and format conversion on the knee joint X-ray image data to generate a standard image with a resolution of 128×128, and then uses the MRIcro toolkit to convert the MRI image data into NIfTI format and set a region of interest mask to retain the pixel values ​​of key areas, and finally obtains clinical evaluation data from the CCKS2017 database through crawler technology, and integrates all preprocessed data into a standardized data set.

[0008] Preferably, the feature automatic learning module in the model building module adopts a double-layer convolutional neural network structure with n×m convolution kernels to extract n×m dimensional feature vectors from multimodal image data, and the potential relationship mining module adopts a Transformer architecture with h attention heads, and realizes multi-dimensional analysis of the relationship between features by setting the query matrix, key matrix and value matrix, and finally merges the analysis results of multi-head attention through the weighted average method.

[0009] Preferably, the model building module adopts two different model architectures according to the number of convolutional layers k. When k=1, a hybrid model architecture is used to use the output of the convolutional neural network as the input of the Transformer. When k>1, the Transformer architecture is used alone and the output results of the feature automatic learning module are input into the multi-head attention mechanism for further analysis.

[0010] Preferably, the model building module realizes the interpretability of the model through the following steps: first, the prediction results are preprocessed by the soft labeling method, then the convolutional layer feature map is back-propagated to obtain the weight gradient, then the weight gradient is multiplied by the soft label to generate an attention map, and finally the attention map is multiplied by the original feature map to obtain the final Grad-CAM visualization result.

[0011] Preferably, the model building module optimizes the model parameters using a loss function including a cross entropy loss term and a regularization term, controls the parameter update speed by setting a learning rate and an update step size, and evaluates the model performance after each round of training to decide whether to continue the optimization.

[0012] Preferably, the system deployment module includes a knee arthritis diagnosis module and a knee arthritis prediction module, wherein the knee arthritis diagnosis module first receives multimodal image input, then calculates the diagnosis probability through a deep learning model, and finally outputs the diagnosis result based on a preset threshold, and the knee arthritis prediction module analyzes the disease development trend based on historical data and outputs the risk level.

[0013] A method for an early diagnosis system of knee arthritis based on multimodal deep learning comprises the following steps: firstly, acquiring and preprocessing multimodal data through the data collection module, then training a deep learning model using the model construction module, and finally deploying the system deployment module to provide online diagnosis services.

[0014] Preferably, the following steps are specifically included: when a user uploads multimodal imaging data for diagnosis, the system first outputs a normal or abnormal diagnosis result through the knee arthritis diagnosis module, and then outputs a low-risk or high-risk prediction result through the knee arthritis prediction module; when the diagnosis result is abnormal or the prediction result is high-risk, the system automatically generates and displays the Grad-CAM visualization result.

[0015] Preferably, when the diagnosis result is normal or the prediction result is low risk, the system displays the diagnosis basis through the following steps: first, calculate and screen the top 10 image features with the highest importance weights, and then display these features in the form of interactive icons, where the user can view the feature description and mark the corresponding area in the original image with a rectangular box when hovering the mouse over the icon. When the diagnosis result is abnormal, the system simultaneously displays the classification result, confidence score and explanatory image.

[0016] The beneficial technical effects brought by the technical solution of the present invention are:

[0017] At the architectural design level, the present invention adopts a modular design approach to form an organic closed loop between the functional modules. The data collection module provides high-quality multimodal data for model training through a standardized preprocessing process. The model building module is based on an innovative hybrid deep learning architecture to achieve deep fusion and intelligent analysis of multimodal data. The system deployment module not only provides convenient online diagnostic services, but also continuously collects clinical feedback to optimize model performance. This closed-loop design ensures that the system can continuously improve itself and adapt to various needs in clinical practice.

[0018] In terms of algorithmic innovation, this invention combines convolutional neural networks with Transformer architecture in a breakthrough way, giving full play to the advantages of the two models. Convolutional networks are good at capturing local image features, while Transformer architecture is good at modeling long-range dependencies. The combination of the two greatly improves the model's feature extraction and relationship modeling capabilities. At the same time, by integrating the improved Grad-CAM visualization method, the system can generate intuitive explanatory results to help doctors understand the diagnostic basis of the model.

[0019] In terms of practicality, the present invention has carried out targeted optimization in multiple key links: First, by setting reasonable diagnostic thresholds and risk levels, the system can provide differentiated diagnostic suggestions for patients in different situations. Second, the innovative interactive result display method allows doctors to understand and verify the diagnostic results more intuitively. Third, the system's continuous optimization mechanism ensures that the diagnostic performance can be continuously improved as data accumulates.

[0020] The technical solution of the present invention not only solves several key problems existing in the prior art, but also shows significant technical effects in practical applications: the diagnostic accuracy is greatly improved, the interpretability is significantly enhanced, and the system usability is significantly improved. In particular, in terms of early lesion identification and disease progression prediction, the present invention shows unique technical advantages and provides strong technical support for improving the diagnosis and treatment of knee arthritis.

[0021] It should be emphasized that there is a close synergistic effect between the various technical features of the present invention. For example, the fusion of multimodal data not only improves the accuracy of diagnosis, but also provides a richer source of information for interpretable analysis. The continuous optimization mechanism of the system further enhances the generalization ability of the model through the continuous accumulation of clinical data. The organic combination of these technical features ultimately forms an intelligent diagnostic system with excellent performance, reliability and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a system logic block diagram of the present invention.

[0023] Figure 2 This is a diagram of the deep learning model architecture of the present invention. DETAILED DESCRIPTION

[0024] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0025] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0026] See also Figure 1-2 The present invention provides an early diagnosis system for knee arthritis based on multimodal deep learning, including a data collection module 1, a model building module 2 and a system deployment module 3. Among them, the data collection module 1 is mainly responsible for acquiring and preprocessing knee joint X-ray image data, MRI image data and clinical evaluation data. The model building module 2 realizes intelligent mapping from multimodal data to clinical evaluation phenotype labels by combining convolutional neural networks and Transformer architecture. The system deployment module 3 is responsible for deploying the diagnosis system and providing online services. Preferably, the system deployment module 3 also has the function of continuously collecting diagnostic data and feeding it back to the data collection module 1, so as to realize continuous optimization of the diagnostic model.

[0027] In a preferred embodiment of the present invention, the specific workflow of the data collection module 1 is as follows: first, a complete multimodal data set including knee joint X-ray images, MRI images and corresponding clinical evaluation data is obtained from the hospital database. Then, the Pydicom toolkit is used to process the X-ray images, which first decodes the DICO M format file and then uniformly converts the image to a standard resolution of 128×128 pixels. In a preferred embodiment of the present invention, the resolution of 128×128 is selected because the size can retain sufficient diagnostic information and effectively control the computational complexity. Then, the MRIcro toolkit is used to process the MRI data and convert it into the standard NIfTI format. It is worth noting that a region of interest mask is set during the conversion process, so that the analysis can be focused on key diagnostic sites such as knee cartilage and meniscus. Finally, standardized clinical evaluation data are obtained from authoritative databases such as CCKS2017 through crawler technology.

[0028] In a preferred embodiment of the present invention, the model building module 2 of the present invention includes two core components: a feature automatic learning submodule 21 and a potential relationship mining submodule 22. The feature automatic learning submodule 21 adopts a double-layer convolutional neural network structure with n×m convolution kernels to extract n×m-dimensional feature vectors from multimodal images. Preferably, n and m are 64 and 128, respectively, which can ensure the adequacy of feature extraction without causing excessive computational burden. The potential relationship mining submodule 22 adopts a Transformer architecture with h attention heads (preferably, h=8), and realizes multi-dimensional analysis of the relationship between features by setting the query matrix Q, the key matrix K and the value matrix V.

[0029] In a preferred embodiment of the present invention, the model building module 2 adopts different model architecture strategies according to the number of convolution layers k. Specifically, when k=1, a hybrid model architecture is adopted, and the output features of the convolutional neural network are directly input into the Transformer for further processing. When k>1, feature extraction and relationship modeling are completely based on the Transformer architecture. The present invention has been verified through a large number of experiments that this adaptive architecture selection strategy can achieve good results in different scenarios.

[0030] In a preferred embodiment of the present invention, the present invention innovatively integrates a visualization method based on Grad-CAM. The specific implementation steps are: first, the prediction results are preprocessed with soft labels, preferably the prediction probability is multiplied by a coefficient of 0.8, so as to avoid the occurrence of zero values ​​affecting subsequent processing. Then, the convolutional layer feature map is back-propagated to obtain the importance weights of each feature map. Then, the weight is multiplied by the soft label to generate an attention map, and finally the attention map is multiplied by the original feature map to obtain the final Grad-CAM visualization result.

[0031] In a preferred embodiment of the present invention, the present invention uses an improved loss function for parameter optimization during the model training process. The loss function includes a cross entropy loss term and an L2 regularization term, wherein the regularization coefficient is preferably set to 0.001. In terms of the parameter update strategy, the Adam optimizer is used for gradient descent, the initial learning rate is set to 0.001, and the learning rate is reduced to the original 0.1 when the performance of the validation set does not improve for 5 consecutive epochs.

[0032] In a preferred embodiment of the present invention, the system deployment module 3 includes two main functional units: a diagnosis submodule 31 and a prediction submodule 32. After receiving the images uploaded by the user, the diagnosis submodule 31 first performs standardized preprocessing, and then inputs the deep learning model to calculate the diagnosis probability. In a preferred embodiment of the present invention, two diagnostic thresholds of 0.3 and 0.7 are set: when the diagnosis probability is greater than 0.7, it is judged as abnormal, when it is less than 0.3, it is judged as normal, and when it is between 0.3 and 0.7, further examination is recommended. The prediction submodule 32 analyzes the risk of disease progression based on historical data, and the high-risk warning threshold is preferably set to 0.8.

[0033] The present invention also discloses a method for an early diagnosis system of knee arthritis based on multimodal deep learning. The specific implementation process of the present invention is divided into three main stages. The first is the data preprocessing stage, in which a complete multimodal data set is acquired and sorted through a data collection module 1. Then there is the model training stage, in which a deep learning model is built and optimized using a model building module 2. Finally, there is the system deployment stage, in which a system deployment module 3 is enabled to provide online diagnostic services.

[0034] In a preferred embodiment of the present invention, when a user needs to diagnose knee arthritis, he or she first uploads relevant images through the system's Web interface. The system will automatically complete image preprocessing and give a normal / abnormal diagnosis result through the diagnosis submodule 31, and output a low-risk / high-risk risk prediction through the prediction submodule 32. In particular, when the system determines that it is abnormal or high-risk, it will automatically call the Grad-CAM module to generate an explanatory visualization result in the form of a heat map.

[0035] In a preferred embodiment of the present invention, the present invention adopts a humanized interactive design in the result display link. For normal or low-risk cases, the system will calculate and screen the top 10 image features with the highest importance weights, and present them in the form of interactive icons. When the user hovers the mouse over a feature icon, the color of the icon will deepen (preferably, from light blue #E6F3FF to dark blue #1890FF), and the corresponding area will be marked with a red rectangular frame (line width is preferably 2 pixels) on the original image, and a detailed text description will be provided. For abnormal cases, in addition to giving clear classification results, confidence scores between 0 and 1 and explanatory images based on Grad-CAM will also be displayed.

[0036] In the embodiment of the present invention, when processing knee joint X-ray images, the data collection module 1, in addition to adjusting the resolution to 128×128 pixels, also performs the following preprocessing steps: first, the image is histogram equalized, preferably using the CLAHE (Contrast Limited Adaptive Histogram Equalization) algorithm, where the contrast limit threshold is set to 3.0 and the grid size is set to 8×8 pixels. Then, the image is denoised, preferably using the Non-local Means filtering algorithm, where the search window size is set to 21×21 pixels, the similarity window size is set to 7×7 pixels, and the filtering strength h is set to 10. This can effectively improve the image quality and facilitate subsequent feature extraction.

[0037] For the preprocessing of MRI data, the present invention uses the MRIcro toolkit to set the ROI mask, and the specific steps are as follows: first, based on anatomical knowledge, the knee joint area is divided into multiple regions of interest such as cartilage area, meniscus area, ligament area, etc. Then, a binary mask is set for each area, where the pixel value of the region of interest is set to 1, and the other regions are set to 0. Preferably, an atlas-based automatic segmentation method is used when performing regional segmentation, wherein the atlas template comes from the public OAI (Osteoarthritis Initiative) data set.

[0038] The specific parameter configuration of the two-layer convolutional neural network in the feature automatic learning submodule 21 is as follows: the first convolution layer uses 64 3×3 convolution kernels, with a step size of 1 and a padding of 1, followed by a ReLU activation function and a 2×2 maximum pooling layer (with a step size of 2). The second convolution layer uses 128 3×3 convolution kernels, and the other parameter configurations are the same as the first layer. A Batch Normalization layer is also inserted between the two convolution layers to help speed up the training process and improve the generalization ability of the model.

[0039] In the latent relationship mining submodule 22, the specific implementation of the Transformer architecture includes: first, the input features are mapped to a 512-dimensional feature space through linear projection, and then multi-head self-attention calculation is performed through 8 attention heads. The feature dimension of each attention head is 64 (i.e., 512 / 8), and the attention dropout rate is set to 0.1. In the feedforward neural network part, a two-layer structure is adopted, the hidden layer dimension is 2048, and the ReLU activation function is also used.

[0040] In the training process of the present invention, an improved learning strategy is adopted: first, the convolutional neural network part is initialized using the pre-trained ImageNet weights, and then a two-stage training method is adopted. In the first stage, the convolutional layer parameters are fixed, and only the Transformer part is trained for 15 epochs; in the second stage, all layers are unfrozen for end-to-end training, and 30 epochs are trained. Preferably, the AUC index is calculated on the validation set after each epoch, and when the validation set AUC no longer improves for 5 consecutive epochs, the learning rate is reduced to the original 0.1.

[0041] In terms of the implementation of the interpretability method, the present invention improves the traditional Grad-CAM method: first, when calculating the category activation map, not only the last layer of convolutional feature maps is considered, but also the feature map information of the second to last layer and the third to last layer is integrated. Specifically, after the three layers of feature maps are upsampled to the same spatial resolution, they are fused by weighted averaging, where the weights are 0.5, 0.3, and 0.2 respectively. In this way, multi-scale saliency information can be obtained, which improves the accuracy of the visualization results.

[0042] In the system deployment phase, the present invention implements a dynamic update mechanism for the model: the system will regularly (preferably once every two weeks) collect new diagnostic data and calculate the performance of the model on these new data. When it is found that the diagnostic accuracy of certain types of cases has dropped significantly (for example, the average accuracy of 10 consecutive cases of the same type is less than 0.8), the system will automatically trigger the model fine-tuning process. During fine-tuning, a smaller learning rate (preferably 0.0001) is used for incremental learning to avoid destroying the good performance of the model on existing data.

[0043] In practical applications, the present invention has found that the above technical solution can well balance diagnostic accuracy and computational efficiency. For example, in a 6-month clinical trial in a tertiary hospital, the system's diagnostic accuracy for more than 2,000 cases of knee arthritis reached 92.3%, with an average single diagnosis time of only 0.8 seconds, and more than 95% of the diagnosis results were recognized by the attending physician.

[0044] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An early diagnosis system for knee arthritis based on multimodal deep learning, characterized by: It includes a data collection module, a model building module and a system deployment module, wherein the data collection module is used to collect and preprocess multimodal data including knee joint X-ray image data, MRI image data and clinical evaluation data; the model building module constructs a deep learning model by combining convolutional neural network and Transformer architecture, and adopts feature automatic learning module and potential relationship mining module to realize the mapping from multimodal data to clinical evaluation phenotype labels, and integrates Grad-CAM interpretability method to understand the basis of model prediction; the system deployment module is responsible for deploying the diagnostic system and providing online services, and continuously collects diagnostic data to feed back to the data collection module for model optimization.

2. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 1 is characterized in that: The data collection module first obtains a multimodal data set including knee joint X-ray image data, MRI image data and corresponding clinical evaluation data from the hospital database, and then uses the Pydicom toolkit to decode and convert the DICOM file of the knee joint X-ray image data to generate a standard image with a resolution of 128×128. The MRIcro toolkit is then used to convert the MRI image data into NIfTI format and set a region of interest mask to retain the pixel values ​​of key areas. Finally, the clinical evaluation data is obtained from the CCKS2017 database through crawler technology, and all preprocessed data are integrated into a standardized data set.

3. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 1, characterized in that: The feature automatic learning module in the model construction module adopts a double-layer convolutional neural network structure with n×m convolution kernels to extract n×m dimensional feature vectors from multimodal image data. The potential relationship mining module adopts a Transformer architecture with h attention heads, and realizes multi-dimensional analysis of the relationship between features by setting the query matrix, key matrix and value matrix, and finally merges the analysis results of multi-head attention through the weighted average method.

4. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 3 is characterized in that: The model building module adopts two different model architectures according to the number of convolutional layers k. When k=1, a hybrid model architecture is used to use the output of the convolutional neural network as the input of the Transformer. When k>1, the Transformer architecture is used alone and the output results of the feature automatic learning module are input into the multi-head attention mechanism for further analysis.

5. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 4 is characterized in that: The model building module achieves the interpretability of the model through the following steps: first, the prediction results are preprocessed by the soft labeling method, then the convolutional layer feature map is back-propagated to obtain the weight gradient, then the weight gradient is multiplied by the soft label to generate the attention map, and finally the attention map is multiplied by the original feature map to obtain the final Grad-CAM visualization result.

6. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 3 is characterized in that: The model building module optimizes the model parameters using a loss function that includes a cross entropy loss term and a regularization term, controls the parameter update speed by setting a learning rate and an update step size, and evaluates the model performance after each round of training to decide whether to continue optimization.

7. The early diagnosis system for knee arthritis based on multimodal deep learning according to claim 1, characterized in that: The system deployment module includes a knee arthritis diagnosis module and a knee arthritis prediction module, wherein the knee arthritis diagnosis module first receives multimodal image input, then calculates the diagnosis probability through a deep learning model, and finally outputs the diagnosis result based on a preset threshold. The knee arthritis prediction module analyzes the disease development trend based on historical data and outputs the risk level.

8. A method for early diagnosis of knee arthritis based on multimodal deep learning according to any one of claims 1 to 7, characterized in that: The method comprises the following steps: firstly, acquiring and preprocessing multimodal data through the data collection module, then training a deep learning model using the model construction module, and finally deploying the system deployment module to provide online diagnostic services.

9. The method according to claim 8, characterized in that The specific steps include: When the user uploads multimodal imaging data for diagnosis, the system first outputs a normal or abnormal diagnosis result through the knee arthritis diagnosis module, and then outputs a low-risk or high-risk prediction result through the knee arthritis prediction module. When the diagnosis result is abnormal or the prediction result is high-risk, the system automatically generates and displays the Grad-CAM visualization result.

10. The method according to claim 9, characterized in that When the diagnosis result is normal or the predicted result is low risk, the system displays the diagnosis basis through the following steps: first, calculate and filter the top 10 image features with the highest importance weights, and then display these features in the form of interactive icons. When the user hovers the mouse over the icon, he can view the feature description and mark the corresponding area in the original image with a rectangular box. When the diagnosis result is abnormal, the system simultaneously displays the classification result, confidence score and explanatory image.

Images