Multi-modal image feature fused knee joint fraction prediction method and system

Abstract

The invention discloses a knee joint fraction prediction method and system fused with multi-modal image features. The knee joint fraction prediction method fused with the multi-modal image features obtains a knee joint prediction fraction of an object through five steps. The method has the beneficial effects that 1, deep learning, radiomics characteristics and combined clinical characteristics are fused, the magnetic resonance T1 sequence image of the knee joint is processed and analyzed, the popliteal artery ROI is marked, and related radiomics and deep network characteristics are extracted; compared with traditional single feature analysis, the method has the advantages that more comprehensive and accurate disease assessment is provided, and the accuracy and reliability of knee osteoarthritis prediction are improved. And 2, the method is based on the obvious relation between the popliteal artery and the knee bone joint, innovatively analyzes the image features of the popliteal artery and combines clinical data, so that the method not only can predict the occurrence and progress of the osteoarthritis more accurately in an earlier stage, but also obviously improves the prediction effect.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing and intelligent medical technology, and in particular to a knee joint score prediction method and system integrating multimodal image features. Background Art

[0002] Osteoarthritis (OA) is the most common musculoskeletal disease, mainly affecting weight-bearing joints such as the hip and knee. Osteoarthritis (KOA) is a chronic disease that affects the knee joint and its surrounding tissues, mainly leading to progressive damage to the articular cartilage, which in turn affects the subchondral bone and surrounding synovial structures. Its symptoms and signs include pain, stiffness, reduced joint mobility and muscle weakness, which seriously affect the subject's quality of life. Existing prediction methods mainly rely on clinical symptoms and imaging examinations, but their accuracy and sensitivity need to be improved.

[0003] Therefore, in view of the shortcomings of the existing technology, it is necessary to provide a knee joint score prediction method and system that integrates multimodal imaging features to solve the shortcomings of the existing technology. Summary of the invention

[0004] The first object of the present invention is to avoid the shortcomings of the prior art and provide a method for predicting knee joint scores by integrating multimodal image features. The method for predicting knee joint scores by integrating multimodal image features can accurately obtain a predicted knee joint score.

[0005] The above-mentioned purpose of the present invention is achieved by the following technical measures:

[0006] A method for predicting knee joint scores by integrating multimodal imaging features is provided, comprising the following steps:

[0007] S1. Obtain magnetic resonance T1 sequence images and clinical characteristics of the target knee joint;

[0008] S2, respectively segmenting the popliteal artery region in the magnetic resonance T1 sequence image obtained in S1;

[0009] S3, extracting the imaging features in the magnetic resonance T1 sequence image after segmenting the popliteal artery region in S2, and then screening the imaging features to obtain the screened imaging features;

[0010] S4, extracting deep learning features from the magnetic resonance T1 sequence image after segmenting the popliteal artery region in S2 through the trained deep learning model, and then screening the deep learning features to obtain screened deep learning features;

[0011] S5. Combine the post-screening radiomics features obtained in S3, the post-screening deep learning features obtained in S4, and the clinical features obtained in S1, and input the combined features into a logistic regression classifier to obtain the subject's knee joint prediction score.

[0012] Preferably, the above-mentioned trained deep learning model is a trained Inception v3 network model.

[0013] Preferably, the training method of the above-mentioned trained deep learning model is performed by the following steps:

[0014] A1. Use PyTorch to load the Inception v3 network model and pre-trained parameters in PyTorch for training, and freeze the network parameters of the Inception v3 network model except the Mixed_7c layer;

[0015] A2. Input the training set into the Inception v3 network model, and the network updates the network parameters of the Mixed_7c layer to obtain the updated Inception v3 network model.

[0016] A3, determine the current number of training times, and go to A4 when the number of training times reaches the target number of training times, otherwise return to A2;

[0017] A4. Define the current updated Inception v3 network model as the trained Inception v3 network model.

[0018] Preferably, the above A2 specifically inputs the training set into the Inception v3 network model, then converts the Inception v3 network model into a binary classification output and maps the output value of the Inception v3 network model to [0,1] according to the sigmoid activation function, and updates the network parameters of the Mixed_7c layer to obtain an updated Inception v3 network model.

[0019] Preferably, the above S3 is obtained by the following steps:

[0020] S3.1. Input the magnetic resonance T1 sequence images after segmentation of the popliteal artery region in S2 into Python3.8, and extract the radiomics features through the Pyradiomicsv3.0 toolkit;

[0021] S3.2. In the R language environment, use the mRMRe package to implement the minimum redundancy maximum correlation algorithm and the glmnet package to implement the Lasso combined with the five-fold cross-validation algorithm to perform multiple repeated screenings of S3.1 to obtain the imaging omics features and obtain the screened imaging omics features.

[0022] Preferably, the above S4 is obtained by the following steps:

[0023] S4.1, inputting the magnetic resonance T1 sequence images obtained after segmentation of the popliteal artery region in S2 into the trained deep learning model to obtain deep learning features;

[0024] S4.2. In the R language environment, use the mRMRe package to implement the minimum redundancy maximum correlation algorithm and the glmnet package to implement Lasso combined with the five-fold cross-validation algorithm to perform multiple repeated screenings in S4.2 to obtain deep learning features and obtain the deep learning features after screening.

[0025] Preferably, the above clinical characteristics are characteristics of the subject's body mass index, characteristics of having undergone surgery or arthroscopy, and characteristics of taking nonsteroidal anti-inflammatory drugs.

[0026] Preferably, the above-mentioned post-screening imaging features consist of 12 first-order features and 17 high-order texture features.

[0027] The 12 first-order features are original_firstorder_Median_1, wavelet.HHH_firstorder_Median_1, wavelet.LLH_firstorder_Kurtosis_1, log.sigma.3. mm.3D_firstorder_Kurtosis_1, log.sigma.2.mm.3D_firstorder_RootMeanSquared_1, wavelet.LHH_firstorder_90Percentile.2, wav elet.LLH_firstorder_InterquartileRange.2, wavelet.LLL_firstorder_90Percentile.2, wavelet.LHH_firstorder_90Percentile.2, log.sigma.2.mm.3D_firstorder_Median.2, wavelet.LHL_firstorder_10Percentile.2 and wavelet.HLL_firstorder_90Percentile.2.

[0028] The 17 high-order texture features are

[0029] log.sigma.2.mm.3D_glcm_InverseVariance_1, original_glcm_Idn_1, original_glcm_MCC_1, log.sigma.3.mm.3D_ngtdm_Busyness_1, wavelet.LHL_glcm_ClusterShade_1, wavelet.LLL_glrlm_Sh ortRunEmphasis_1, wavelet.HLL_glcm_DifferenceVariance_1, original_gldm_LargeDependenceLowGrayLevelEmphasis_1, log.sigma.2.mm.3D_glcm_Idm_1, wavelet.HHL_glcm_ClusterShade.2,

[0030] wavelet.HHL_glcm_Idn.2, log.sigma.3.mm.3D_glcm_Idm1.2, log.sigma.5.mm.3D_ngtdm_Contrast.2, wavelet.HLL_glcm_ClusterShade.2,

[0031] log.sigma.2.mm.3D_ngtdm_Busyness.2, log.sigma.2.mm.3D_glszm_ZoneVariance.2 and

[0032] wavelet.HLH_glcm_MCC.2.

[0033] Preferably, the above-screened deep learning features are the 608th parameter features, 1211th parameter features, 1093th parameter features, 852th parameter features, 1241st parameter features, 1107th parameter features, 922nd parameter features, 1897th parameter features, 1978th ​​parameter features, 467th parameter features, 528th parameter features, 909th parameter features, 1 388th parameter feature, 941st parameter feature, 1932nd parameter feature, 773rd parameter feature, 1336th parameter feature, 647th parameter feature, 1103rd parameter feature, 853rd parameter feature, 590th parameter feature, 391st parameter feature, 1593rd parameter feature, 60th parameter feature, 192nd parameter feature, 1500th parameter feature, 1177th parameter feature and 954th parameter feature.

[0034] Preferably, the above-mentioned imaging omics features are shape features, first-order statistical features and high-order texture features.

[0035] The second object of the present invention is to avoid the shortcomings of the prior art and provide a knee joint score prediction system integrating multimodal image features. The knee joint score prediction system integrating multimodal image features can accurately obtain the predicted score of the knee joint.

[0036] The above-mentioned purpose of the present invention is achieved by the following technical measures:

[0037] Provided is a knee joint score prediction system integrating multimodal imaging features, using the above-mentioned knee joint score prediction method integrating multimodal imaging features;

[0038] The knee joint score prediction system integrating multimodal imaging features of the present invention is provided with:

[0039] An acquisition unit—acquires a magnetic resonance T1 sequence image of the knee joint of the object to be tested and predicted;

[0040] Radiomics feature extraction unit: extracts radiomics features from the magnetic resonance T1 sequence images after segmenting the popliteal artery region;

[0041] Deep learning feature extraction unit: extracts deep learning features from the magnetic resonance T1 sequence images after segmentation of the popliteal artery area sign through the trained deep learning model;

[0042] Radiomics feature screening unit: screening the radiomics features in the radiomics feature extraction unit to obtain the screened radiomics features;

[0043] Deep learning feature screening unit - screening the deep learning features in the deep learning feature extraction unit to obtain the screened deep learning features;

[0044] Classification unit: Merge the trained deep learning model, the screened deep learning features and the clinical features, input the merged features into the logistic regression classifier to obtain the knee joint prediction score of the subject.

[0045] The present invention provides a method and system for predicting a knee joint score by integrating multimodal imaging features, wherein the method for predicting a knee joint score by integrating multimodal imaging features comprises the following steps: S1, obtaining a magnetic resonance T1 sequence image and clinical features of a target knee joint; S2, respectively segmenting the popliteal artery regions in the magnetic resonance T1 sequence image obtained by S1; S3, extracting radiomic features in the magnetic resonance T1 sequence image after the popliteal artery region is segmented in S2, and then screening the radiomic features to obtain screened radiomic features; S4, extracting deep learning features in the magnetic resonance T1 sequence image after the popliteal artery region is segmented in S2 through a trained deep learning model, and then screening the deep learning features to obtain screened deep learning features; S5, merging the screened radiomic features obtained by S3, the screened deep learning features obtained by S4, and the clinical features obtained by S1, and inputting the merged features into a logistic regression classifier to obtain a predicted knee joint score of the target. The beneficial effects of the present invention are: 1. By integrating deep learning, imaging genomics features and clinical features, the magnetic resonance T1 sequence images of the knee joint are processed and analyzed, the popliteal artery ROI is marked, and relevant imaging genomics and deep network features are extracted; compared with the traditional single feature analysis, the multimodal data fusion method of the present invention provides a more comprehensive and accurate disease assessment, and improves the accuracy and reliability of knee osteoarthritis prediction. 2. The present invention is based on the significant connection between the popliteal artery and the knee joint, and innovatively develops a new prediction method by analyzing the imaging features of the popliteal artery and combining clinical data. This method can not only predict the occurrence and progression of osteoarthritis earlier and more accurately, but also significantly improves the prediction effect. DETAILED DESCRIPTION

[0046] The technical solution of the present invention is further described in conjunction with the following embodiments.

[0047] Example 1

[0048] A method for predicting knee joint scores by integrating multimodal imaging features comprises the following steps:

[0049] S1. Obtaining magnetic resonance T1 sequence images and clinical characteristics of the subject's knee joint; wherein the clinical characteristics are the subject's body mass index characteristics, characteristics of having undergone surgery or arthroscopy, and characteristics of taking nonsteroidal anti-inflammatory drugs;

[0050] S2, respectively segmenting the popliteal artery region in the magnetic resonance T1 sequence image obtained in S1;

[0051] S3, extracting the imaging features in the magnetic resonance T1 sequence image after segmenting the popliteal artery region in S2, and then screening the imaging features to obtain the screened imaging features; wherein the imaging features are shape features, first-order statistical features, and high-order texture features;

[0052] S4, extracting deep learning features from the magnetic resonance T1 sequence image after segmentation of the popliteal artery area sign in S2 through the trained deep learning model, and then screening the deep learning features to obtain screened deep learning features;

[0053] S5. Merge the post-screening radiomics features obtained in S3, the post-screening deep learning features obtained in S4, and the clinical features obtained in S1, and input the merged features into a logistic regression classifier to obtain the subject's knee joint prediction score.

[0054] It should be noted that the feature merging in S5 of the present invention refers to merging the post-screening radiomics features, post-screening deep learning features and clinical features into the same row in an Excel table.

[0055] In S2 of the present invention, the magnetic resonance T1 sequence images are imported into ITK-SNAP software, and the popliteal artery region (ROI) of the image is manually delineated and segmented.

[0056] Among them, S3 is obtained by the following steps:

[0057] S3.1. Input the magnetic resonance T1 sequence images after segmentation of the popliteal artery region in S2 into Python3.8, and extract the radiomics features through the Pyradiomicsv3.0 toolkit;

[0058] S3.2. In the R language environment, use the mRMRe package to implement the minimum redundancy maximum correlation algorithm and the glmnet package to implement the Lasso combined with the five-fold cross-validation algorithm to perform multiple repeated screenings of S3.1 to obtain the imaging omics features and obtain the screened imaging omics features.

[0059] Among them, S4 is obtained by the following steps:

[0060] S4.1, inputting the magnetic resonance T1 sequence images obtained after segmentation of the popliteal artery region in S2 into the trained deep learning model to obtain deep learning features;

[0061] S4.2. In the R language environment, use the mRMRe package to implement the minimum redundancy maximum correlation algorithm and the glmnet package to implement Lasso combined with the five-fold cross-validation algorithm to perform multiple repeated screenings in S4.2 to obtain deep learning features and obtain the deep learning features after screening.

[0062] The screening of radiomics features and deep learning features in the present invention first uses the minimum redundancy maximum correlation (mRMR) for preliminary screening to maximize the correlation between features and target variables while minimizing the redundancy between features. Then, the minimum absolute value convergence and selection operator (LASSO) and cross validation (CV) are used to further screen features.

[0063] LASSO formula:

[0064]

[0065] Among them, y i is the actual value of the target variable, x i is the feature vector of the i-th sample, β is the weight coefficient of the model (the parameter to be optimized), and λ is the hyperparameter of the regularization strength (controlling the complexity of the model). LASSO introduces the L1 regularization term Constrain the model parameters so that the coefficients β of some features j is pushed towards zero, thus automatically selecting important features and eliminating unimportant ones.

[0066] It should be noted that the mRMR algorithm (Max-Relevance and Min-Redundancy) is to find a set of features in the original feature set that have the greatest correlation with the final output result (Max-Relevance), but the smallest correlation between the features (Min-Redundancy). Therefore, in order to maximize the correlation between the features and the target variable and minimize the redundancy between the features, a line of data_mrmr<-mRMR.data(data=mydatasp) code is directly used to implement feature screening. For example: After the algorithm, 1000 features are screened to only 20, there will be no redundancy between these 20 features, that is, there will be no similar features. If two similar features appear, only one needs to be retained. These 20 features are all most correlated with the target variable (whether it is knee osteoarthritis), that is, the top 20 with the highest correlation. The features screened in this way are beneficial to building a machine learning classifier to predict knee osteoarthritis.

[0067] Then, LASSO (least absolute convergence and selection operator algorithm) is used to screen the features selected by the mRMR algorithm again, but this screening is different from the mRMR algorithm screening. Specifically, a sparse model is output, and only a few features have coefficients that are not zero. In other words, the Lasso method will reduce the weights of features that have little impact on the prediction of the target variable to zero, thereby achieving feature selection. Lasso is also implemented in one line of code in R language model_lasso<-glmnet(tmp_x,tmp_y,family="binomial",nlambda=50,alpha=1,standardize=TRUE).

[0068] The trained deep learning model is a trained Inception v3 network model. The training method of the trained deep learning model is performed by the following steps:

[0069] A1. Use PyTorch to load the Inception v3 network model and pre-trained parameters in PyTorch for training, and freeze all network parameters except the Mixed_7c layer for training. The training process will be completed on the PyTorch (Torch 2.3.1, CUDA 11.7) deep learning framework.

[0070] A2. Input the training set into the Inception v3 network model, and the network updates the network parameters of the Mixed_7c layer to obtain the updated Inception v3 network model.

[0071] A3, determine the current number of training times, and go to A4 when the number of training times reaches the target number of training times, otherwise return to A2;

[0072] A4. Define the current updated Inception v3 network model as the trained Inception v3 network model.

[0073] It should be noted that PyTorch is an underlying framework for deep learning, similar to a tool. All network definitions, training, and verifications of the present invention are completed on pytorch. Moreover, pytorch comes with many networks and their pre-trained parameters. With the underlying framework of pytorch, there is no need to define networks and other tasks separately. Therefore, the Inception v3 network model of the present invention and the pre-trained parameters in PyTorch can be directly loaded from pytorch.

[0074] A2 specifically inputs the training set into the Inception v3 network model, then converts the Inception v3 network model into a binary output and maps the output value of the Inception v3 network model to [0,1] according to the sigmoid activation function, and updates the network parameters of the Mixed_7c layer to obtain the updated Inception v3 network model.

[0075] It should be noted that the Inception v3 network model that comes with pytorch is a pre-trained Inception v3 network model. In this embodiment, 666 magnetic resonance T1 sequence images and corresponding real labels are used for training, where the 666 cases are divided into a ratio of 7:3, that is, the number of training set cases is 466 and the number of test set cases is 200. 466 magnetic resonance T1 sequence images and corresponding real labels are input into the Inception v3 network model, and the predicted probability of the output object and the real label of the object are calculated by the binary cross entropy loss function, and the non-frozen parameters are updated in reverse to achieve the fine-tuning effect, and the trained Inception v3 network model is obtained. The target number of training times in this embodiment is 500 times.

[0076] The imaging features after screening in the present invention are composed of 12 first-order features and 17 high-order texture features. Among them, the 12 first-order features are

[0077] original_firstorder_Median_1、

[0078] wavelet.HHH_firstorder_Median_1,

[0079] wavelet.LLH_firstorder_Kurtosis_1, log.sigma.3.mm.3D_firstorder_Kurtosis_1, log.sigma.2.mm.3D_firstorder_RootMeanSquared_1, wavelet.LHH_firstorder_90Percentile.2, wavelet.LLH_firsto rder_InterquartileRange.2, wavelet.LLL_firstorder_90Percentile.2, wavelet.LHH_firstorder_90Percentile.2, log.sigma.2.mm.3D_firstorder_Median.2, wavelet.LHL_firstorder_10Percentile.2 and

[0080] wavelet.HLL_firstorder_90Percentile.2。

[0081] The 17 high - order texture features are

[0082] log.sigma.2.mm.3D_glcm_InverseVariance_1, original_glcm_Idn_1, original_glcm_MCC_1, log.sigma.3.mm.3D_ngtdm_Busyness_1, wavelet.LHL_glcm_ClusterShade_1,

[0083] wavelet.LLL_glrlm_ShortRunEmphasis_1, wavelet.HLL_glcm_DifferenceVariance_1, original_gldm_LargeDependenceLowGrayLevelEmphasis_1, log.sigma.2.mm.3D_glcm_Idm_1,

[0084] wavelet.HHL_glcm_ClusterShade.2,

[0085] wavelet.HHL_glcm_Idn.2, log.sigma.3.mm.3D_glcm_Idm1.2, log.sigma.5.mm.3D_ngtdm_Contrast.2, wavelet.HLL_glcm_ClusterShade.2,

[0086] log.sigma.2.mm.3D_ngtdm_Busyness.2, log.sigma.2.mm.3D_glszm_ZoneVariance.2 and

[0087] wavelet.HLH_glcm_MCC.2。

[0088] The screened deep learning features of the present invention are the 608th parameter feature, the 1211th parameter feature, the 1093rd parameter feature, the 852nd parameter feature, the 1241st parameter feature, the 1107th parameter feature, the 922nd parameter feature, the 1897th parameter feature, the 1978th ​​parameter feature, the 467th parameter feature, the 528th parameter feature, the 909th parameter feature, the 13 88th parameter feature, 941st parameter feature, 1932nd parameter feature, 773rd parameter feature, 1336th parameter feature, 647th parameter feature, 1103rd parameter feature, 853rd parameter feature, 590th parameter feature, 391st parameter feature, 1593rd parameter feature, 60th parameter feature, 192nd parameter feature, 1500th parameter feature, 1177th parameter feature and 954th parameter feature.

[0089] It should be noted that the blood supply to the knee joint is essential for maintaining its normal function and health. The popliteal artery is the main blood supply artery to the knee joint, providing abundant blood supply to the joint and its surrounding tissues. Studies have shown that changes in blood flow in the popliteal artery are closely related to the pathological process of knee osteoarthritis. As an emerging technology for precise evaluation, radiomics uses artificial intelligence to objectively and quantitatively analyze and evaluate heterogeneous information in medical images. Through the magnetic resonance T1 sequence images of the knee joint, the region of interest (ROI) in the popliteal artery area is marked, and the radiomics features of the popliteal artery are extracted. Combining the deep features extracted by deep learning with clinical features, a model for accurately predicting osteoarthritis is constructed. This multimodal fusion prediction method improves the accuracy and sensitivity of the prediction.

[0090] The beneficial effects of the knee joint score prediction method that integrates multimodal imaging features are as follows: 1. By integrating deep learning, imaging genomics features and clinical features, the magnetic resonance T1 sequence images of the knee joint are processed and analyzed, the popliteal artery ROI is marked, and relevant imaging genomics and deep network features are extracted; compared with the traditional single feature analysis, the multimodal data fusion method of the present invention provides a more comprehensive and accurate disease assessment, and improves the accuracy and reliability of knee osteoarthritis prediction. 2. The present invention is based on the significant connection between the popliteal artery and the knee joint, and innovatively develops a new prediction method by analyzing the imaging features of the popliteal artery and combining clinical data. This method can not only predict the occurrence and progression of osteoarthritis earlier and more accurately, but also significantly improve the prediction effect.

[0091] Example 2

[0092] A method for predicting knee joint scores by integrating multimodal imaging features, other features are the same as those in Example 1. In this example, 666 magnetic resonance T1 sequence images and corresponding real labels are used for training, wherein the 666 examples are divided into a ratio of 7:3, that is, the number of training set examples is 466 and the number of test set examples is 200. The 466 magnetic resonance T1 sequence images and the corresponding real labels are input into the Inception v3 network model, the predicted probability of the output object and the real label of the object are calculated by the binary cross entropy loss function, and the non-frozen parameters are updated in reverse to achieve the fine-tuning effect, and the trained Inception v3 network model is obtained. The target number of training times in this example is 500 times.

[0093] In this embodiment, a total of 2446-dimensional imaging features (also known as radiomic features) are extracted in S3.1, including first-order features, shape-based features, and high-order texture features. In the R language (version 4.3.0) environment, the mRMRe package and the glmnet package are used to implement the minimum redundancy maximum correlation and Lasso combined with five-fold cross validation for 50 repetitions to select key features, that is, after screening, there are 29 imaging features, including 12 first-order features and 17 high-order texture features.

[0094] The 12 first-order features and 17 high-order texture features were weighted by the regression coefficient of Lasso combined with the five-fold cross-validation algorithm through the glmnet package to calculate the radiomics scoring formula. The feature names and coefficients of each screened radiomics feature are shown in Table 1. Among them, the radiomics score = Intercept + (F1 × 0.84737) + (F2 × -1.56565) + (F3 × 0.86044) + (F4 × 0.78123) + (F5 × -0.37287) + (F6 × -0.86426) + (F7 × 2.07435) + (F8 × -1.86193) + (F9 × -1.30815) + (F10 × -0.47243) + (F11 × -0.10174) + (F12 × -0.07069) + (H1 × 0.87889) + (H2 × 1.91672) + (H3 × -3. (H14×-0.48365)+(H15×0.66395)+(H16×1.81836)+(H17×-1.25403), Intercept is the bias term.

[0095] Table 1. Feature names and coefficients of radiomics features after screening

[0096]

[0097] In this embodiment, a total of 2050-dimensional deep image features are extracted in S4.1. In the R language (version 4.3.0), the mRMRe package and the glmnet package are used to implement the minimum redundancy maximum correlation and Lasso combined with five-fold cross validation for 50 repetitions to select key features, that is, a total of 28 deep learning features are selected after screening.

[0098] 28 features are selected through the algorithm. The number of features selected can be different by adjusting the hyperparameters of the algorithm. For example, if the features with a correlation greater than 90% are changed to features with a correlation greater than 95%, the number of features selected will be different. The features of this embodiment are changed to have a correlation of 95%, and a total of 28 deep features are selected from 2048 deep features. The deep learning scoring formula, where raw_data608 is the 608th deep learning feature among the 2048 deep learning features, and raw_data1093 is the 1093rd deep learning feature among the 2048 deep learning features. The deep learning score = 0.5016862-1.3592160×raw_data608-1.1122070×

[0099] raw_data1211+1.0141249×raw_data1093-1.2208380×raw_data852-0.1761829×raw_data124 1-1.5264384×raw_data1107+1.2612275×raw_data922-1.2413815×raw_data1897-0.8177814×

[0100] raw_data1978+0.6624622×raw_data467+0.6044658×raw_data528-0.6168924×raw_data909 +0.5313487×raw_data1388+0.2362643×raw_data941-0.9070500×raw_data1932+0.9004163×

[0101] raw_data773+1.8661871×raw_data1336+1.7596755×

[0102] raw_data647+0.6650059×raw_data1103+0.4627288×raw_data853-1.6372479×raw_data59 0-0.3395007×raw_data391-1.7718665×raw_data1593-0.1818095×raw_data60+1.3656279×

[0103] raw_data192+0.5499795×raw_data1500-1.2056339×raw_data1177-0.3269084×raw_data954.

[0104] The clinical characteristics analysis in this study was performed in R (version 4.3.0). For the total samples of the training set and the test set, the t test or Mann-Whitney U test (according to the data distribution) was used to compare continuous variables, and the chi-square test (χ2 test) was used to compare categorical variables. Table 2 shows the clinical baseline characteristics of all patients.

[0105] Table 2 is a clinical feature baseline table, which shows a statistical analysis of clinical features. The method used for clinical feature screening of the present invention is univariate Logistic regression analysis and multivariate Logistic regression analysis, and the analysis results are shown in Table 3. The present invention first uses univariate Logistic regression analysis to screen one by one, and considers the relationship between multiple independent variables (such as continuous variables, categorical variables, etc.) and dependent variables while using multivariate Logistic regression analysis to control the interference of other factors. First, univariate analysis is performed to screen potential related variables, and then the independent influence of variables is confirmed by multivariate analysis to screen out clinical variables, and finally the clinical features of the present invention are determined.

[0106] Table 2. Clinical baseline table

[0107]

[0108]

[0109] Table 3 shows the results of univariate and multivariate logistic regression analysis of clinical variables associated with knee osteoarthritis (KOA) in the training set. Variables significantly associated with KOA were screened out (all P values ​​< 0.05). The selected clinical characteristics included body mass index (BMI), whether knee surgery or arthroscopy had been performed (yes or no), and whether nonsteroidal anti-inflammatory drugs (NSAIDs) were taken (yes or no).

[0110] Table 3. Univariate and multivariate logistic regression analysis

[0111]

[0112] This example evaluates three different models: clinical model, clinical + radiomics model, and clinical + deep features + radiomics model (the present invention). The prediction performance differences between these models were compared by DeLong test, as shown in Table 4.

[0113] Table 4 Differences in predictive performance of clinical model, clinical + radiomics model, and clinical + deep features + radiomics model

[0114]

[0115] The results showed that all prediction models were verified in the test cohort, and the AUC values ​​of the clinical model, clinical + imaging omics model, and the present invention in the training / test set were 0.618 / 0.552, 0.735 / 0.680, and 0.791 / 0.738, respectively. It can be seen that the present invention exhibited good predictive efficacy in the test cohort. In the Delong test in the last column of Table 3, there was a significant statistical difference between the present invention and the other two models (P<0.01). AUC = 0.5: The performance of the model is equivalent to random guessing; AUC < 0.5: The performance of the model is worse than random guessing; AUC = 1.0: The model can perfectly distinguish between positive and negative samples.

[0116] The clinical + deep feature + imaging omics model of the present invention showed significant differences (p < 0.05) compared with the model using clinical features alone and the clinical + imaging omics model. This shows that by introducing a comprehensive model of deep features and imaging omics features, the accuracy of the knee joint prediction score can be significantly improved. In the specific implementation, the clinical, deep and omics features extracted and screened based on the clinical data of the subject and the knee joint MRI images were collected. The clinical model, clinical + imaging omics model and clinical + deep feature + imaging omics model were constructed, and the DeLong test was used to verify the significant differences between the models to ensure that the final knee joint prediction score has efficient and reliable prediction capabilities.

[0117] Example 3

[0118] A knee joint score prediction system integrating multimodal image features adopts the knee joint score prediction method integrating multimodal image features of embodiment 1 or embodiment 2.

[0119] The knee joint score prediction system integrating multimodal imaging features is configured as follows:

[0120] An acquisition unit—acquires a magnetic resonance T1 sequence image of the knee joint of the object to be tested and predicted;

[0121] Radiomics feature extraction unit: extracts radiomics features from the magnetic resonance T1 sequence images after segmenting the popliteal artery region;

[0122] Deep learning feature extraction unit: extracts deep learning features from the magnetic resonance T1 sequence images after segmentation of the popliteal artery area sign through the trained deep learning model;

[0123] Radiomics feature screening unit: screening the radiomics features in the radiomics feature extraction unit to obtain the screened radiomics features;

[0124] Deep learning feature screening unit - screening the deep learning features in the deep learning feature extraction unit to obtain the screened deep learning features;

[0125] Classification unit: Merge the trained deep learning model, the screened deep learning features and the clinical features, input the merged features into the logistic regression classifier to obtain the knee joint prediction score of the subject.

[0126] The beneficial effects of the knee joint score prediction system that integrates multimodal imaging features of the present invention are: 1. By integrating deep learning, imaging genomics features and clinical features, the magnetic resonance T1 sequence images of the knee joint are processed and analyzed, the popliteal artery ROI is marked, and relevant imaging genomics and deep network features are extracted; compared with the traditional single feature analysis, the multimodal data fusion method of the present invention provides a more comprehensive and accurate disease assessment, and improves the accuracy and reliability of knee osteoarthritis prediction. 2. The present invention is based on the significant connection between the popliteal artery and the knee joint, and innovatively develops a new prediction method by analyzing the imaging features of the popliteal artery and combining clinical data. This method can not only predict the occurrence and progression of osteoarthritis earlier and more accurately, but also significantly improves the prediction effect.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.

Claims

1. A method for predicting knee joint scores by integrating multimodal image features, characterized in that: The steps include: S1. Obtain magnetic resonance T1 sequence images and clinical characteristics of the target knee joint; S2, respectively segmenting the popliteal artery region in the magnetic resonance T1 sequence image obtained in S1; S3, extracting the imaging features in the magnetic resonance T1 sequence image after segmenting the popliteal artery region in S2, and then screening the imaging features to obtain the screened imaging features; S4, extracting deep learning features from the magnetic resonance T1 sequence image after segmenting the popliteal artery region in S2 through the trained deep learning model, and then screening the deep learning features to obtain screened deep learning features; S5. Merge the post-screening radiomics features obtained in S3, the post-screening deep learning features obtained in S4, and the clinical features obtained in S1, and input the merged features into a logistic regression classifier to obtain the subject's knee joint prediction score.

2. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 1, characterized in that: The trained deep learning model is a trained Inception v3 network model; The training method of the trained deep learning model is performed by the following steps: A1. Use PyTorch to load the Inception v3 network model and pre-trained parameters in PyTorch for training, and freeze the network parameters of the Inception v3 network model except the Mixed_7c layer; A2. Input the training set into the Inception v3 network model, and the network updates the network parameters of the Mixed_7c layer to obtain the updated Inception v3 network model. A3, determine the current number of training times, and go to A4 when the number of training times reaches the target number of training times, otherwise return to A2; A4. Define the current updated Inception v3 network model as the trained Inception v3 network model.

3. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 2, characterized in that: Specifically, A2 inputs the training set into the Inception v3 network model, then converts the Inception v3 network model into a binary classification output, maps the output value of the Inception v3 network model to [0, 1] according to the sigmoid activation function, and updates the network parameters of the Mixed_7c layer to obtain an updated Inception v3 network model.

4. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 3, characterized in that: The S3 is obtained by the following steps: S3.

1. Input the magnetic resonance T1 sequence images after segmentation of the popliteal artery region in S2 into Python3.8, and extract the radiomics features through the Pyradiomicsv3.0 toolkit; S3.

2. In the R language environment, use the mRMRe package to implement the minimum redundancy maximum correlation algorithm and the glmnet package to implement the Lasso combined with the five-fold cross-validation algorithm to perform multiple repeated screenings of S3.1 to obtain the imaging omics features and obtain the screened imaging omics features.

5. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 4, characterized in that: The S4 is obtained by the following steps: S4.1, inputting the magnetic resonance T1 sequence images obtained after segmentation of the popliteal artery region in S2 into the trained deep learning model to obtain deep learning features; S4.

2. In the R language environment, use the mRMRe package to implement the minimum redundancy maximum correlation algorithm and the glmnet package to implement Lasso combined with the five-fold cross-validation algorithm to perform multiple repeated screenings in S4.2 to obtain deep learning features and obtain the deep learning features after screening.

6. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 1, characterized in that: The clinical characteristics include the subject's body mass index, surgery or arthroscopy, and nonsteroidal anti-inflammatory drug use.

7. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 1, characterized in that: The post-screening radiomics features consist of 12 first-order features and 17 high-order texture features; The 12 first-order features are original_firstorder_Median_1, wavelet.HHH_firstorder_Median_1, wavelet.LLH_firstorder_Kurtosis_1, log.sigma.3.mm.3D_firstorder_Kurtosis_1、 log.sigma.2.mm.3D_firstorder_RootMeanSquared_1, wavelet.LHH_firstorder_90Percentile.2, wavelet.LLH_firstorder_InterquartileRange.2, wavelet.LLL_firstorder_90Percentile.2, wavelet.LHH_firstorder_90Percentile.2, log.sigma.2.mm.3D_firstorder_Median.2、 wavelet.LHL_firstorder_10Percentile.2 and wavelet.HLL_firstorder_90Percentile.2; The 17 high-order texture features are log.sigma.2.mm.3D_glcm_InverseVariance_1, original_glcm_Idn_1, original_glcm_MCC_1, log.sigma.3.mm.3D_ngtdm_Busyness_1、 wavelet.LHL_glcm_ClusterShade_1, wavelet.LLL_glrlm_ShortRunEmphasis_1, wavelet.HLL_glcm_DifferenceVariance_1, original_gldm_LargeDependenceLowGrayLevelEmphasis_1, log.sigma.2.mm.3D_glcm_Idm_1, wavelet.HHL_glcm_ClusterShade.2, wavelet.HHL_glcm_Idn.2, log.sigma.3.mm.3D_glcm_Idm1.2, log.sigma.5.mm.3D_ngtdm_Contrast.2, wavelet.HLL_glcm_ClusterShade.2, log.sigma.2.mm.3D_ngtdm_Busyness.2、 log.sigma.2.mm.3D_glszm_ZoneVariance.2 and wavelet.HLH_glcm_MCC.

2.

8. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 1, characterized in that: The screened deep learning features are the 608th parameter feature, the 1211th parameter feature, the 1093th parameter feature, the 852th parameter feature, the 1241st parameter feature, the 1107th parameter feature, the 922nd parameter feature, the 1897th parameter feature, the 1978th ​​parameter feature, the 467th parameter feature, the 528th parameter feature, the 909th parameter feature, the 138 8 parameter features, 941st parameter features, 1932nd parameter features, 773rd parameter features, 1336th parameter features, 647th parameter features, 1103th parameter features, 853rd parameter features, 590th parameter features, 391st parameter features, 1593rd parameter features, 60th parameter features, 192nd parameter features, 1500th parameter features, 1177th parameter features and 954th parameter features.

9. The method for predicting knee joint scores by integrating multimodal imaging features according to claim 1, characterized in that: The imaging omics features are shape features, first-order statistical features and high-order texture features.

10. A knee joint score prediction system integrating multimodal image features, characterized in that: A method for predicting knee joint scores by fusing multimodal imaging features as described in any one of claims 1 to 9; The settings are: An acquisition unit—acquires a magnetic resonance T1 sequence image of the knee joint of the object to be tested and predicted; Radiomics feature extraction unit: extracts radiomics features from the magnetic resonance T1 sequence images after segmenting the popliteal artery region; Deep learning feature extraction unit: extracts deep learning features from the magnetic resonance T1 sequence images after segmentation of the popliteal artery area sign through the trained deep learning model; Radiomics feature screening unit: screening the radiomics features in the radiomics feature extraction unit to obtain the screened radiomics features; Deep learning feature screening unit - screening the deep learning features in the deep learning feature extraction unit to obtain the screened deep learning features; Classification unit: Merge the trained deep learning model, the screened deep learning features and the clinical features, input the merged features into the logistic regression classifier to obtain the knee joint prediction score of the subject.

Images