AI combined MRI and clinical JIA diagnosis system and storage medium

Abstract

The invention belongs to the technical field of intelligent diagnosis, and particularly relates to an AI combined MRI and clinical JIA diagnosis system and a storage medium. According to the system disclosed by the invention, the early auxiliary diagnosis of the juvenile idiopathic arthritis is carried out by combining artificial intelligence with multi-dimensional and multi-modal information of multi-sequence MRI images of knee joints of children and various clinical information. The main technology of the method is child knee joint tissue segmentation based on deep learning, a multi-dimensional feature extraction strategy based on a segmentation result, and disease classification based on multi-modal feature integration and deep learning. By integrating the multi-dimensional features of the multi-sequence MRI images and fusing different modal features such as image information and clinical information, an auxiliary diagnosis result with high accuracy can be provided. The technology provided by the invention is beneficial to the realization of early diagnosis and early treatment of juvenile idiopathic arthritis, and has a very good application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent diagnosis, and particularly relates to a JIA diagnosis system combining AI, MRI and clinical information, and a storage medium. Background Art

[0002] Juvenile idiopathic arthritis (JIA) is the most common rheumatic disease in children, characterized by chronic arthritis, with multiple subtypes, and the knee joint is most commonly affected. The cause of this disease is unknown, the diagnosis lacks specificity, and there is no radical cure method. Long-term chronic inflammation can lead to growth retardation, joint deformity and even disability in children with JIA, affecting the future physical and mental health development of children; some subtypes have severe conditions, and in addition to joint symptoms, they can also cause damage to other organs, and the prognosis is poor, ultimately leading to death. Therefore, early diagnosis and treatment of JIA are very necessary.

[0003] Magnetic resonance imaging (MRI) is the best imaging diagnostic method for JIA. Traditional sequences can show synovial thickening, bone marrow edema, cartilage and ligament damage, and joint effusion. Emerging 3D-ZTE, MAGIC, and CEST sequences can perform quantitative analysis and functional evaluation on the above manifestations, and can also show subchondral bone destruction. However, at present, this kind of imaging evaluation is greatly affected by subjective factors of the diagnostician: it is closely related to the ability and experience of the diagnostic physician; manually delineating the region of interest for quantitative and functional analysis is also restricted by the medical staff's understanding of the knee joint anatomy of children, the principles and basic knowledge of MRI, and the cognitive level of JIA disease, and the repeatability is not high. The above problems will affect the accuracy of early diagnosis of JIA to varying degrees, and then affect subsequent treatment. Therefore, it is urgent to introduce new methods to achieve precise diagnosis and treatment of JIA.

[0004] Artificial intelligence (AI) is the latest scientific and technological achievement at present. Through data mining, feature extraction, and building personalized models, it has been widely applied in various fields. With the application progress of machine learning and deep learning in the field of medical image analysis, various semi-automatic / full-automatic tissue segmentation methods and quantitative evaluation methods have been continuously proposed, which have greatly promoted the development of computer-aided diagnosis systems and made up for the deficiencies of manual image interpretation. In the prior art, the application of AI in the musculoskeletal system mainly focuses on adult degenerative osteoarticular changes such as osteoarthritis, and there is also some research on the damage of knee joint cartilage and cruciate ligaments; in the children's osteoarticular system, it is currently mainly used for bone age determination, fracture evaluation, and diagnosis of leg length difference, and the application in MRI cartilage, ligaments, and effusion is rare. At present, computer-aided diagnosis systems based on images have made certain progress in rheumatoid arthritis and osteoarthritis.

[0005] However, these existing artificial intelligence methods and systems still cannot be applied to the auxiliary diagnosis of JIA. This is because the affected population of JIA is children, with an age distribution ranging from 6 to 16 years old. Like other structures, the knee joint is also in a continuous growth and development stage. The epiphyses and metaphyses of the femur, tibia, and patella have different morphologies and compositions at different ages. Therefore, the artificial intelligence diagnosis of children's musculoskeletal system-related diseases is more difficult than that of adults, and it is difficult to achieve accurate diagnosis with a single algorithm and model. Summary of the Invention

[0006] Aiming at the problems of the existing technology, the present invention provides a JIA diagnosis system and a storage medium that combine AI, MRI, and clinical data.

[0007] A JIA diagnosis system that combines AI, MRI, and clinical data includes: An input module configured to input multi-sequence MRI images of the knee joint of a subject, demographic information, and clinical examination information; A tissue segmentation module configured to segment the multi-sequence MRI images of the knee joint to obtain regions of interest corresponding to the distal femur cartilage, proximal tibia cartilage, patella cartilage, anterior cruciate ligament, posterior cruciate ligament, biceps femoris, and effusion; An MRI image feature extraction module configured to extract multi-dimensional image features of each tissue structure from the regions of interest, where the multi-dimensional image features include radiomics features, quantitative features, deep learning local and global features; A text feature extraction module configured to extract the demographic information, clinical information, and blood test information through an NLP model to obtain text features; A feature fusion module configured to fuse the multi-dimensional image features and text features to obtain fusion features; A prediction module configured to input the fusion features into a classification model to obtain a prediction result of juvenile idiopathic arthritis.

[0008] Preferably, the age of the subject is 6 - 16 years old.

[0009] Preferably, the sequences of the multi-sequence MRI images include the following three sequences: T1WI, T2WI, PDWI, and at least one of the following sequences: 3D-ZTE, MAGIC, CEST.

[0010] Preferably, the demographic information includes the following information: gender, age, BMI; the clinical information includes: medical history, which knee joint (left knee or right knee); the blood test information includes at least one of the following information: rheumatism screening, autoantibodies, cytokines, humoral immunity, HLA-B27, gene test results; Among them, the rheumatism screening includes at least one of RF, CRP, ASO, the autoantibodies include at least one of ANA, RNP, SSB, the cytokines include at least one of TNF, IFN, IL-1, IL-2, IL-2R, IL-4, IL-6, IL-8, IL-10, and the humoral immunity includes at least one of IgG, IgA, IgM, IgE, C1, C3, C4.

[0011] Preferably, the radiomics features include at least one of the following features: first-order statistical features, two-dimensional shape-based features, three-dimensional shape-based features, adjacent gray level difference matrix features, gray level run length matrix features, gray level co-occurrence matrix features, gray level dependence matrix features, and gray level size zone matrix features; The quantitative features include at least one of the following features: tissue volume, cartilage thickness, signal distribution features.

[0012] Preferably, the deep learning local and global features are obtained according to the following steps: Step 1, data preprocessing: Convert the region of interest image from a 3D image to a 2D image; Select the display maximum layer of the region of interest image as the target, and extract the upper and lower layer images of this layer to form 3 channels as the global image to be processed; Crop the extracted 3-channel image to obtain the maximum bounding box of the region of interest; scale the cropped image to a fixed size as the local image to be processed; Normalize the global image and the local image according to the maximum and minimum values of the images; Step 2, feature extraction: Input the preprocessed global and local 2D images into the adjusted ResNet-18 and VGG-16 pre-trained models respectively for feature extraction; the adjusted ResNet-18 or VGG-16 removes the fully connected layer to obtain the global feature and the local feature respectively.

[0013] Preferably, in the feature fusion module, feature fusion is performed through a Transformer encoder.

[0014] Preferably, in the feature fusion module, the steps for feature fusion include: Step A: Concatenate the multi-dimensional image features into a vector sequence, and input the vector sequence into a Transformer encoder. Through the self-attention mechanism, capture the relationships between the input multi-dimensional image features; Step B: Use a set of learnable tokens with adjustable quantity as the working memory of the neural network, and input the text features into the working memory through the cross-attention mechanism; Step C: Then use a Transformer encoder to merge the output tokens of Step A and Step B to achieve feature fusion of the two modalities.

[0015] Preferably, in the prediction module, the algorithm of the classification model is selected from a multi-layer perceptron.

[0016] The present invention also provides a computer-readable storage medium, on which there is stored: a computer program for implementing the above-mentioned AI combined with MRI and clinical JIA diagnosis system.

[0017] The present invention provides an intelligent diagnosis method and system capable of early diagnosing juvenile idiopathic arthritis. By combining artificial intelligence with multi-sequence MRI image information and clinical information, and specifically selecting the included features according to the characteristics of children's knee joints, an algorithm for feature extraction and fusion is constructed, achieving accurate prediction and providing strong guarantee for subsequent treatment.

[0018] Compared with the existing computer-aided diagnosis and treatment systems, the present invention gives full play to the features provided by MRI images, from imaging genomics to deep learning features combined with quantitative information, ensuring that the content of the images is not masked by other information in the multi-modal model. On the other hand, the previous AI musculoskeletal systems only focused on the features of the images themselves and established a single model, which is not applicable to the growing children group. To address this problem, in addition to constructing a deep learning segmentation model, an imaging genomics feature extraction model, and a local deep learning model by annotating 7 structures including 3 cartilages, 2 ligaments, 1 muscle, and joint effusion in the knee joints of JIA children, the present application also considered the image information not included in the annotation, such as bone marrow edema, synovial thickening, and bone erosion, and constructed a global deep learning model, and combined the above models to construct an auxiliary diagnosis model for multi-dimensional image features. The present application also fuses the image features and clinical features to construct a multi-modal AI diagnosis model for JIA. In addition, the present application screens out the information related to JIA from numerous clinical information as text features to avoid the interference of disease-unrelated features in the medical record information on the multi-modal model. These improvements jointly overcome the problem that the existing intelligent diagnosis methods for joint diseases are not applicable to the children group. In summary, the present invention introduces AI technology and, based on the existing auxiliary diagnosis system, innovatively constructs a multi-dimensional and multi-modal auxiliary diagnosis system based on multi-sequence MRI images and clinical information for the characteristics of children's knee joints and the disease characteristics of JIA, achieving early and precise diagnosis of JIA and having good application prospects.

[0019] Obviously, based on the above content of the present invention, according to the common general knowledge and customary means in the art, without departing from the above basic technical idea of the present invention, various other forms of modifications, substitutions or changes can be made.

[0020] The following further detailed description of the above content of the present invention is given in the form of specific embodiments by way of examples. However, this should not be construed as limiting the scope of the above subject matter of the present invention to the following examples. All technologies implemented based on the above content of the present invention fall within the scope of the present invention. Brief Description of the Drawings

[0021] Figure 1 It is a schematic flow chart of Embodiment 1. Detailed Description of the Specific Embodiment

[0022] It should be noted specifically that the algorithms for steps such as data acquisition, transmission, storage, and processing not specifically described in the embodiments, as well as the hardware structures and circuit connections not specifically described, can be implemented through the content publicly disclosed in the prior art.

[0023] Embodiment 1 AI-Combined MRI and Clinical Juvenile Idiopathic Arthritis Diagnosis System The system of this embodiment is as Figure 1 shown and includes: An input module configured to input multi-sequence MRI images of the knee joint of a subject, demographic information, and clinical examination information; A tissue segmentation module configured to segment the multi-sequence MRI images of the knee joint to obtain regions of interest corresponding to the distal femur cartilage, proximal tibia cartilage, patellar cartilage, anterior cruciate ligament, posterior cruciate ligament, biceps femoris, and effusion; An MRI image feature extraction module configured to extract multi-dimensional image features of each tissue structure from the regions of interest, where the multi-dimensional image features include radiomics features, quantitative features, deep learning local and global features; A text feature extraction module configured to extract the demographic information, clinical information, and blood test information through an NLP model to obtain text features; A feature fusion module configured to fuse the multi-dimensional image features and text features to obtain fusion features; A prediction module, configured to input the fused features into a classification model to obtain a prediction result of juvenile idiopathic arthritis.

[0024] Among them, The system of this embodiment is applicable to subjects aged 6 - 16 years old.

[0025] The steps for predicting juvenile idiopathic arthritis based on the system of this embodiment include: (1) Input the multi - sequence MRI images, demographic information, and clinical examination information of the subject.

[0026] Among the input information, the sequences of the multi - sequence MRI images include T1WI, T2WI, PDWI, 3D - ZTE, MAGIC, CEST. The demographic information includes gender, age, BMI; the clinical information includes medical history, which knee joint (left knee or right knee); the blood test information includes: rheumatism screening, autoantibodies, cytokines, humoral immunity, HLA - B27, gene test results.

[0027] Among them, the rheumatism screening includes RF, CRP, ASO, the autoantibodies include ANA, RNP, SSB, the cytokines include TNF, IFN, IL - 1, IL - 2, IL - 2R, IL - 4, IL - 6, IL - 8, IL - 10, and the humoral immunity includes IgG, IgA, IgM, IgE, C1, C3, C4.

[0028] (2) Knee joint tissue segmentation In this embodiment, the segmentation process and the training process of the segmentation model include: 1) Pre - annotation: Import the MRI image data of all sequences into the annotation platform for semi - automatic and manual annotation, and sequentially annotate the regions of interest such as the distal femoral cartilage, proximal tibial cartilage, patellar cartilage, anterior cruciate ligament, posterior cruciate ligament, biceps femoris, and effusion. The annotation results are modified by two radiologists using the annotation platform software, and reviewed by another senior radiologist. If there are significant differences in the standard results, the senior doctor will arbitrate. The unqualified annotation data needs to be re - annotated.

[0029] 2) Image pre - processing: Adopt data pre - processing, mainly including data format conversion, image cropping, resampling, and standardization.

[0030] 3) Based on the annotated data, train the deep - learning network on the linux system (Ubuntu), and evaluate the performance according to the average Dice similarity coefficient of 5 - fold cross - validation during the training process.

[0031] 4) Use the nnUnet tissue segmentation model that meets the accuracy requirements to process the unlabeled image data and obtain the tissue segmentation results of the children's knee joints.

[0032] (3) Feature extraction In this embodiment, the process of feature extraction and the training process of the model include: 1) Radiomics features: Based on the specific tissue structure regions extracted by the segmentation model, analyze the radiomics features of different tissue structures respectively, and combine these features to form a feature matrix. Resample the image data to a voxel size of 1x1x1 mm 3 ; Use the PyRadiomics software package to normalize the signal intensity of the MRI image to [0, 255]; Use the PyRadiomics software package to automatically extract various radiomics features of each tissue from the segmentation result mask, such as first-order statistical features of various tissues, two-dimensional shape-based features, three-dimensional shape-based features, adjacent gray level difference matrix features, gray level run length matrix features, gray level co-occurrence matrix features, gray level dependence matrix features, and gray level size zone matrix features.

[0033] 2) Tissue quantitative features: Perform surface modeling on the aforementioned children's knee joint tissues, and statistically analyze the quantitative features of different tissue structures, including the sum of the volumes of voxels completely located inside the object as the tissue volume, estimate the cartilage thickness using the Euclidean distance between the bone-cartilage interface and the cartilage-surrounding tissue interface, calculate the signal distribution features (histogram, slope, kurtosis, etc.) within different tissue regions, and perform normalization calculations on the quantitative values based on the patient's age, height, and weight information.

[0034] 3) Deep learning local and global features: Use convolutional neural networks to extract local and global deep learning features. In this embodiment, the ResNet-18 network architecture and the VGG-16 network architecture are adopted, pre-trained using ImageNet, and the fully connected layers are removed. Here, it is necessary to convert the 3D input image into 2D, so select the maximum display layer of the image of the region of interest as the target, and at the same time extract the upper and lower layer images to form 3 channels. Crop the extracted 3-channel image to obtain the maximum bounding box of the region of interest, scale it to a fixed size, and then perform normalization processing on the image. Input the processed input image into the ResNet-18 and VGG-16 pre-trained models respectively. After the convolutional layer, use the global max pooling operation to extract the maximum value of the feature map; the local features refer to analyzing the image of the key region separately using a convolutional network, and the global features refer to inputting the original image into the convolutional network for analysis.

[0035] 4) Text features: After cleaning, deduplicating, and handling missing values and outliers for demographic information, clinical information, and blood test information, the processed data is input into a pre-trained large NLP model to extract text features. Specifically, gender, age, BMI, medical history, which knee joint is involved, rheumatic screening in blood test results (RF, CRP, ASO), the autoantibodies (ANA, RNP, SSB), the cytokines (TNF, IFN, IL-1, IL-2, IL-2R, IL-4, IL-6, IL-8, IL-10), the humoral immunity (IgG, IgA, IgM, IgE, C1, C3, C4), HLA-B27, gene test results, etc. are organized into a table. The data is cleaned, deduplicated, missing values are handled, outliers are processed, etc. The processed clinical information features are organized in matrix form and after tokenization, they are input into the pre-trained bert model. After being encoded by the bert model, they are used as text features.

[0036] (4) Feature fusion and prediction In this embodiment, the process of feature extraction and the training process of the model include: Using a multi-modal model to fuse the above-mentioned radiomics features, quantitative features, deep learning local and global features (these multi-dimensional imaging features) and the text features of clinical information together, and using the attention mechanism to establish connections between multi-modal information. The fused feature matrix is used to predict whether the patient meets the diagnostic criteria of JIA.

[0037] Specifically, this embodiment makes the following classification based on the transformer encoder.

[0038] 1) Concatenate the multi-dimensional imaging features into a vector sequence, input the vector sequence into the transformer encoder, and through the self-attention mechanism, capture the relationships between the input multi-dimensional imaging features.

[0039] 2) Use a set of adjustable number of learnable tokens as the working memory of the neural network, and input the text features into the working memory through the cross-attention mechanism.

[0040] 3) Then use another transformer encoder to merge the output tokens of steps 1) and 2) to achieve feature fusion of the two modalities.

[0041] 4) Use the output result of feature fusion of the two modalities in step 3) as the input of the next layer, and use a multi-layer perceptron for classification prediction. Based on the classification output result, evaluate the classification performance, draw the ROC curve, and calculate indicators such as the area under the ROC curve AUC, sensitivity, specificity, precision, recall rate, F1-score, Dice coefficient, etc.

[0042] The technical solution of the present invention will be further described through experiments below.

[0043] I. Experimental method The prediction performance of the method of Example 1 and other control methods with adjusted key tissue and organ types for segmentation was compared in this experiment. The specific experimental group settings are as follows: Experimental group: Execute according to the method of Example 1, that is, the key tissues and organs to be segmented include: distal femoral cartilage, proximal tibial cartilage, patellar cartilage, anterior cruciate ligament, posterior cruciate ligament, biceps femoris, and effusion; Control group 1: The key tissues and organs to be segmented are adjusted to: the whole femur, tibia, patella, quadriceps femoris, medial and lateral collateral ligaments, and other steps (such as feature extraction and feature fusion network) are the same as those in Example 1; Control group 2: The feature fusion network is set as a traditional fully connected network, the key tissues and organs to be segmented remain unchanged, and other steps (such as feature extraction) are the same as those in Example 1.

[0044] II. Experimental results Table 1 Prediction performance of different methods for JIA From the above results, it can be seen that after optimizing the key tissue and organ types and the feature fusion network targeted by the method, the performance of the method of the present invention has been significantly improved in both sensitivity and specificity, and for the first time, a method and system capable of accurately predicting JIA have been realized.

[0045] From the above embodiments, it can be seen that the present invention constructs an early prediction method and system for juvenile idiopathic arthritis, which can obtain accurate prediction results, which is beneficial to the timely treatment of juvenile idiopathic arthritis and has good application prospects.

Claims

1. An AI combined with MRI and clinical JIA diagnosis system, characterized in that, Comprising: An input module, configured to input multi-sequence MRI images of the knee joint of a subject, demographic information, and clinical examination information; A tissue segmentation module, configured to segment the multi-sequence MRI images of the knee joint to obtain regions of interest corresponding to the distal femoral cartilage, proximal tibial cartilage, patellar cartilage, anterior cruciate ligament, posterior cruciate ligament, biceps femoris, and joint effusion; An MRI image feature extraction module, configured to extract multi-dimensional image features of each tissue structure from the regions of interest, where the multi-dimensional image features include radiomics features, quantitative features, and deep learning local and global features; A text feature extraction module, configured to extract the demographic information, clinical information, and blood test information through an NLP model to obtain text features; A feature fusion module, configured to fuse the multi-dimensional image features and text features to obtain fused features; A prediction module, configured to input the fused features into a classification model to obtain a prediction result for juvenile idiopathic arthritis.

2. The AI combined with MRI and clinical JIA diagnosis system according to claim 1, characterized in that: The age of the subject is 6-16 years old.

3. The AI combined with MRI and clinical JIA diagnosis system according to claim 1, wherein: The sequences of the multi-sequence MRI images include the following three sequences: T1WI, T2WI, PDWI, and at least one of the following sequences: 3D-ZTE, MAGIC, CEST.

4. The AI combined with MRI and clinical JIA diagnosis system according to claim 1, wherein: The demographic information includes the following information: gender, age, BMI; the clinical information includes: medical history, which knee joint specifically; the blood test information includes at least one of the following information: rheumatism screening, autoantibodies, cytokines, humoral immunity, HLA-B27, gene test results; Among them, the rheumatism screening includes at least one of RF, CRP, ASO, the autoantibodies include at least one of ANA, RNP, SSB, the cytokines include at least one of TNF, IFN, IL-1, IL-2, IL-2R, IL-4, IL-6, IL-8, IL-10, and the humoral immunity includes at least one of IgG, IgA, IgM, IgE, C1, C3, C4.

5. The AI combined with MRI and clinical JIA diagnosis system according to claim 1, wherein: The radiomics features include at least one of the following features: first-order statistical features, shape-based two-dimensional features, shape-based three-dimensional features, adjacent gray level difference matrix features, gray level run length matrix features, gray level co-occurrence matrix features, gray level dependence matrix features, and gray level size zone matrix features; The quantitative features include at least one of the following features: tissue volume, cartilage thickness, signal distribution features.

6. The AI combined with MRI and clinical JIA diagnosis system according to claim 1, wherein: The deep learning local and global features are obtained according to the following steps: Step 1, data preprocessing: Convert the region of interest image from a 3D image to a 2D image; Select the display maximum layer of the region of interest image as the target, and extract the upper and lower layer images of this layer to form 3 channels as the global image to be processed; Crop the extracted 3-channel image to obtain the maximum bounding box of the region of interest; Scale the cropped image to a fixed size as the local image to be processed; Normalize the global image and the local image according to the maximum and minimum values of the images; Step 2, feature extraction: The preprocessed global and local 2D images are respectively input into the adjusted ResNet-18 and VGG-16 pre-trained models for feature extraction; the fully connected layers are removed from the adjusted ResNet-18 or VGG-16 to obtain global features and local features respectively.

7. The AI combined with MRI and clinical JIA diagnosis system according to claim 1, wherein: In the feature fusion module, feature fusion is performed through a Transformer encoder.

8. The AI combined with MRI and clinical JIA diagnosis system according to claim 7, characterized in that: In the feature fusion module, the steps for feature fusion include: Step A, the multi-dimensional image features are concatenated into a vector sequence, and the vector sequence is input into the Transformer encoder. Through the self-attention mechanism, the relationships between the input multi-dimensional image features are captured; Step B, a set of learnable tokens with adjustable quantity are used as the working memory of the neural network, and the text features are input into the working memory through the cross-attention mechanism; Step C, another Transformer encoder is used to merge the output tokens of Step A and Step B to achieve feature fusion of the two modalities.

9. The AI combined with MRI and clinical JIA diagnosis system according to claim 1, characterized in that: In the prediction module, the algorithm of the classification model is selected from a multi-layer perceptron.

10. A computer-readable storage medium, characterized in that, Stored thereon is a computer program for implementing the AI combined with MRI and clinical JIA diagnosis system according to any one of claims 1-9.

Images